2025-02-15 03:04:00,917 - training_args.py:2100 - _setup_devices - INFO - PyTorch: setting up devices 2025-02-15 03:04:01,457 - configuration_utils.py:731 - _get_config_dict - INFO - loading configuration file ./checkpoints/longvu_llama3_2/config.json 2025-02-15 03:04:01,460 - configuration_utils.py:800 - from_dict - INFO - Model config CambrianConfig { "_name_or_path": "/tmp/iopath_cache/manifold_cache/tree/users/shenx/finetune/09281004-cambrian_llama3_2_t576_ov", "architectures": [ "CambrianLlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "connect_layer": 2, "connector_depth": 3, "connector_only": true, "dino_threshold": 0.83, "drop_threshold": 0.8, "eos_token_id": [ 128001, 128008, 128009 ], "frame_pos": false, "freeze_mm_mlp_adapter": false, "hidden_act": "silu", "hidden_size": 3072, "highres": true, "highres_connect": false, "image_aspect_ratio": "pad", "image_position": 91, "image_token_len": 144, "initializer_range": 0.02, "intermediate_size": 8192, "is_image_newline": true, "is_st_sampler": false, "lowres_token": 8, "max_position_embeddings": 131072, "mlp_bias": false, "mm_patch_merge_type": "flat", "mm_projector_lr": null, "mm_projector_type": "sva", "mm_use_im_patch_token": false, "mm_use_im_start_end": false, "mm_vision_sampler_lr": null, "mm_vision_select_feature": "patch", "mm_vision_select_layer": -2, "mm_vision_tower_aux_list": [ "siglip/CLIP-ViT-SO400M-14-384", "facebook/dinov2-giant-res378" ], "mm_vision_tower_aux_token_len_list": [ 576, 576 ], "mm_vision_tower_lr": null, "model_type": "cambrian_llama", "num_attention_heads": 24, "num_hidden_layers": 28, "num_key_value_heads": 8, "num_of_vision_sampler_layers": 10, "num_query_group": 1, "pretraining_tp": 1, "query_num_list": [ 144 ], "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 32.0, "high_freq_factor": 4.0, "low_freq_factor": 1.0, "original_max_position_embeddings": 8192, "rope_type": "llama3" }, "rope_theta": 500000.0, "spmd_debug": null, "spmd_fsdp_sharding": null, "spmd_mesh": null, "start_of_vision_sampler_layers": 0, "stride_of_vision_sampler_layers": 3, "tie_word_embeddings": false, "tokenizer_model_max_length": 8192, "tokenizer_padding_side": "right", "torch_dtype": "float32", "transformers_version": "4.43.1", "tune_mm_mlp_adapter": false, "unfreeze_mm_vision_tower": false, "use_cache": false, "use_mm_proj": true, "vision_hidden_size": 1024, "vision_tower_aux_token_len_list": [ 576, 576 ], "vocab_size": 128256 } 2025-02-15 03:04:01,461 - modeling_utils.py:3618 - from_pretrained - INFO - loading weights file ./checkpoints/longvu_llama3_2/pytorch_model.bin 2025-02-15 03:04:01,501 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": [ 128001, 128008, 128009 ], "use_cache": false } 2025-02-15 03:04:01,711 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-15 03:04:01,713 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-15 03:04:03,024 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing CambrianLlamaForCausalLM. 2025-02-15 03:04:03,025 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of CambrianLlamaForCausalLM were initialized from the model checkpoint at ./checkpoints/longvu_llama3_2. If your task is similar to the task the model of the checkpoint was trained on, you can already use CambrianLlamaForCausalLM for predictions without further training. 2025-02-15 03:04:03,030 - configuration_utils.py:991 - from_pretrained - INFO - loading configuration file ./checkpoints/longvu_llama3_2/generation_config.json 2025-02-15 03:04:03,031 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": [ 128001, 128008, 128009 ], "temperature": 0.6, "top_p": 0.9 } 2025-02-15 03:04:03,228 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer.json 2025-02-15 03:04:03,228 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file added_tokens.json 2025-02-15 03:04:03,228 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file special_tokens_map.json 2025-02-15 03:04:03,228 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer_config.json 2025-02-15 03:04:03,616 - tokenization_utils_base.py:2533 - _from_pretrained - INFO - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 2025-02-15 03:04:03,995 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/config.json 2025-02-15 03:04:03,998 - configuration_utils.py:800 - from_dict - INFO - Model config SiglipVisionConfig { "attention_dropout": 0.0, "hidden_act": "gelu_pytorch_tanh", "hidden_size": 1152, "image_size": 384, "intermediate_size": 4304, "layer_norm_eps": 1e-06, "model_type": "siglip_vision_model", "num_attention_heads": 16, "num_channels": 3, "num_hidden_layers": 27, "patch_size": 14, "transformers_version": "4.43.1" } 2025-02-15 03:04:03,998 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/model.safetensors 2025-02-15 03:04:04,264 - modeling_utils.py:4440 - _load_pretrained_model - INFO - Some weights of the model checkpoint at google/siglip-so400m-patch14-384 were not used when initializing SiglipVisionModel: ['logit_bias', 'logit_scale', 'text_model.embeddings.position_embedding.weight', 'text_model.embeddings.token_embedding.weight', 'text_model.encoder.layers.0.layer_norm1.bias', 'text_model.encoder.layers.0.layer_norm1.weight', 'text_model.encoder.layers.0.layer_norm2.bias', 'text_model.encoder.layers.0.layer_norm2.weight', 'text_model.encoder.layers.0.mlp.fc1.bias', 'text_model.encoder.layers.0.mlp.fc1.weight', 'text_model.encoder.layers.0.mlp.fc2.bias', 'text_model.encoder.layers.0.mlp.fc2.weight', 'text_model.encoder.layers.0.self_attn.k_proj.bias', 'text_model.encoder.layers.0.self_attn.k_proj.weight', 'text_model.encoder.layers.0.self_attn.out_proj.bias', 'text_model.encoder.layers.0.self_attn.out_proj.weight', 'text_model.encoder.layers.0.self_attn.q_proj.bias', 'text_model.encoder.layers.0.self_attn.q_proj.weight', 'text_model.encoder.layers.0.self_attn.v_proj.bias', 'text_model.encoder.layers.0.self_attn.v_proj.weight', 'text_model.encoder.layers.1.layer_norm1.bias', 'text_model.encoder.layers.1.layer_norm1.weight', 'text_model.encoder.layers.1.layer_norm2.bias', 'text_model.encoder.layers.1.layer_norm2.weight', 'text_model.encoder.layers.1.mlp.fc1.bias', 'text_model.encoder.layers.1.mlp.fc1.weight', 'text_model.encoder.layers.1.mlp.fc2.bias', 'text_model.encoder.layers.1.mlp.fc2.weight', 'text_model.encoder.layers.1.self_attn.k_proj.bias', 'text_model.encoder.layers.1.self_attn.k_proj.weight', 'text_model.encoder.layers.1.self_attn.out_proj.bias', 'text_model.encoder.layers.1.self_attn.out_proj.weight', 'text_model.encoder.layers.1.self_attn.q_proj.bias', 'text_model.encoder.layers.1.self_attn.q_proj.weight', 'text_model.encoder.layers.1.self_attn.v_proj.bias', 'text_model.encoder.layers.1.self_attn.v_proj.weight', 'text_model.encoder.layers.10.layer_norm1.bias', 'text_model.encoder.layers.10.layer_norm1.weight', 'text_model.encoder.layers.10.layer_norm2.bias', 'text_model.encoder.layers.10.layer_norm2.weight', 'text_model.encoder.layers.10.mlp.fc1.bias', 'text_model.encoder.layers.10.mlp.fc1.weight', 'text_model.encoder.layers.10.mlp.fc2.bias', 'text_model.encoder.layers.10.mlp.fc2.weight', 'text_model.encoder.layers.10.self_attn.k_proj.bias', 'text_model.encoder.layers.10.self_attn.k_proj.weight', 'text_model.encoder.layers.10.self_attn.out_proj.bias', 'text_model.encoder.layers.10.self_attn.out_proj.weight', 'text_model.encoder.layers.10.self_attn.q_proj.bias', 'text_model.encoder.layers.10.self_attn.q_proj.weight', 'text_model.encoder.layers.10.self_attn.v_proj.bias', 'text_model.encoder.layers.10.self_attn.v_proj.weight', 'text_model.encoder.layers.11.layer_norm1.bias', 'text_model.encoder.layers.11.layer_norm1.weight', 'text_model.encoder.layers.11.layer_norm2.bias', 'text_model.encoder.layers.11.layer_norm2.weight', 'text_model.encoder.layers.11.mlp.fc1.bias', 'text_model.encoder.layers.11.mlp.fc1.weight', 'text_model.encoder.layers.11.mlp.fc2.bias', 'text_model.encoder.layers.11.mlp.fc2.weight', 'text_model.encoder.layers.11.self_attn.k_proj.bias', 'text_model.encoder.layers.11.self_attn.k_proj.weight', 'text_model.encoder.layers.11.self_attn.out_proj.bias', 'text_model.encoder.layers.11.self_attn.out_proj.weight', 'text_model.encoder.layers.11.self_attn.q_proj.bias', 'text_model.encoder.layers.11.self_attn.q_proj.weight', 'text_model.encoder.layers.11.self_attn.v_proj.bias', 'text_model.encoder.layers.11.self_attn.v_proj.weight', 'text_model.encoder.layers.12.layer_norm1.bias', 'text_model.encoder.layers.12.layer_norm1.weight', 'text_model.encoder.layers.12.layer_norm2.bias', 'text_model.encoder.layers.12.layer_norm2.weight', 'text_model.encoder.layers.12.mlp.fc1.bias', 'text_model.encoder.layers.12.mlp.fc1.weight', 'text_model.encoder.layers.12.mlp.fc2.bias', 'text_model.encoder.layers.12.mlp.fc2.weight', 'text_model.encoder.layers.12.self_attn.k_proj.bias', 'text_model.encoder.layers.12.self_attn.k_proj.weight', 'text_model.encoder.layers.12.self_attn.out_proj.bias', 'text_model.encoder.layers.12.self_attn.out_proj.weight', 'text_model.encoder.layers.12.self_attn.q_proj.bias', 'text_model.encoder.layers.12.self_attn.q_proj.weight', 'text_model.encoder.layers.12.self_attn.v_proj.bias', 'text_model.encoder.layers.12.self_attn.v_proj.weight', 'text_model.encoder.layers.13.layer_norm1.bias', 'text_model.encoder.layers.13.layer_norm1.weight', 'text_model.encoder.layers.13.layer_norm2.bias', 'text_model.encoder.layers.13.layer_norm2.weight', 'text_model.encoder.layers.13.mlp.fc1.bias', 'text_model.encoder.layers.13.mlp.fc1.weight', 'text_model.encoder.layers.13.mlp.fc2.bias', 'text_model.encoder.layers.13.mlp.fc2.weight', 'text_model.encoder.layers.13.self_attn.k_proj.bias', 'text_model.encoder.layers.13.self_attn.k_proj.weight', 'text_model.encoder.layers.13.self_attn.out_proj.bias', 'text_model.encoder.layers.13.self_attn.out_proj.weight', 'text_model.encoder.layers.13.self_attn.q_proj.bias', 'text_model.encoder.layers.13.self_attn.q_proj.weight', 'text_model.encoder.layers.13.self_attn.v_proj.bias', 'text_model.encoder.layers.13.self_attn.v_proj.weight', 'text_model.encoder.layers.14.layer_norm1.bias', 'text_model.encoder.layers.14.layer_norm1.weight', 'text_model.encoder.layers.14.layer_norm2.bias', 'text_model.encoder.layers.14.layer_norm2.weight', 'text_model.encoder.layers.14.mlp.fc1.bias', 'text_model.encoder.layers.14.mlp.fc1.weight', 'text_model.encoder.layers.14.mlp.fc2.bias', 'text_model.encoder.layers.14.mlp.fc2.weight', 'text_model.encoder.layers.14.self_attn.k_proj.bias', 'text_model.encoder.layers.14.self_attn.k_proj.weight', 'text_model.encoder.layers.14.self_attn.out_proj.bias', 'text_model.encoder.layers.14.self_attn.out_proj.weight', 'text_model.encoder.layers.14.self_attn.q_proj.bias', 'text_model.encoder.layers.14.self_attn.q_proj.weight', 'text_model.encoder.layers.14.self_attn.v_proj.bias', 'text_model.encoder.layers.14.self_attn.v_proj.weight', 'text_model.encoder.layers.15.layer_norm1.bias', 'text_model.encoder.layers.15.layer_norm1.weight', 'text_model.encoder.layers.15.layer_norm2.bias', 'text_model.encoder.layers.15.layer_norm2.weight', 'text_model.encoder.layers.15.mlp.fc1.bias', 'text_model.encoder.layers.15.mlp.fc1.weight', 'text_model.encoder.layers.15.mlp.fc2.bias', 'text_model.encoder.layers.15.mlp.fc2.weight', 'text_model.encoder.layers.15.self_attn.k_proj.bias', 'text_model.encoder.layers.15.self_attn.k_proj.weight', 'text_model.encoder.layers.15.self_attn.out_proj.bias', 'text_model.encoder.layers.15.self_attn.out_proj.weight', 'text_model.encoder.layers.15.self_attn.q_proj.bias', 'text_model.encoder.layers.15.self_attn.q_proj.weight', 'text_model.encoder.layers.15.self_attn.v_proj.bias', 'text_model.encoder.layers.15.self_attn.v_proj.weight', 'text_model.encoder.layers.16.layer_norm1.bias', 'text_model.encoder.layers.16.layer_norm1.weight', 'text_model.encoder.layers.16.layer_norm2.bias', 'text_model.encoder.layers.16.layer_norm2.weight', 'text_model.encoder.layers.16.mlp.fc1.bias', 'text_model.encoder.layers.16.mlp.fc1.weight', 'text_model.encoder.layers.16.mlp.fc2.bias', 'text_model.encoder.layers.16.mlp.fc2.weight', 'text_model.encoder.layers.16.self_attn.k_proj.bias', 'text_model.encoder.layers.16.self_attn.k_proj.weight', 'text_model.encoder.layers.16.self_attn.out_proj.bias', 'text_model.encoder.layers.16.self_attn.out_proj.weight', 'text_model.encoder.layers.16.self_attn.q_proj.bias', 'text_model.encoder.layers.16.self_attn.q_proj.weight', 'text_model.encoder.layers.16.self_attn.v_proj.bias', 'text_model.encoder.layers.16.self_attn.v_proj.weight', 'text_model.encoder.layers.17.layer_norm1.bias', 'text_model.encoder.layers.17.layer_norm1.weight', 'text_model.encoder.layers.17.layer_norm2.bias', 'text_model.encoder.layers.17.layer_norm2.weight', 'text_model.encoder.layers.17.mlp.fc1.bias', 'text_model.encoder.layers.17.mlp.fc1.weight', 'text_model.encoder.layers.17.mlp.fc2.bias', 'text_model.encoder.layers.17.mlp.fc2.weight', 'text_model.encoder.layers.17.self_attn.k_proj.bias', 'text_model.encoder.layers.17.self_attn.k_proj.weight', 'text_model.encoder.layers.17.self_attn.out_proj.bias', 'text_model.encoder.layers.17.self_attn.out_proj.weight', 'text_model.encoder.layers.17.self_attn.q_proj.bias', 'text_model.encoder.layers.17.self_attn.q_proj.weight', 'text_model.encoder.layers.17.self_attn.v_proj.bias', 'text_model.encoder.layers.17.self_attn.v_proj.weight', 'text_model.encoder.layers.18.layer_norm1.bias', 'text_model.encoder.layers.18.layer_norm1.weight', 'text_model.encoder.layers.18.layer_norm2.bias', 'text_model.encoder.layers.18.layer_norm2.weight', 'text_model.encoder.layers.18.mlp.fc1.bias', 'text_model.encoder.layers.18.mlp.fc1.weight', 'text_model.encoder.layers.18.mlp.fc2.bias', 'text_model.encoder.layers.18.mlp.fc2.weight', 'text_model.encoder.layers.18.self_attn.k_proj.bias', 'text_model.encoder.layers.18.self_attn.k_proj.weight', 'text_model.encoder.layers.18.self_attn.out_proj.bias', 'text_model.encoder.layers.18.self_attn.out_proj.weight', 'text_model.encoder.layers.18.self_attn.q_proj.bias', 'text_model.encoder.layers.18.self_attn.q_proj.weight', 'text_model.encoder.layers.18.self_attn.v_proj.bias', 'text_model.encoder.layers.18.self_attn.v_proj.weight', 'text_model.encoder.layers.19.layer_norm1.bias', 'text_model.encoder.layers.19.layer_norm1.weight', 'text_model.encoder.layers.19.layer_norm2.bias', 'text_model.encoder.layers.19.layer_norm2.weight', 'text_model.encoder.layers.19.mlp.fc1.bias', 'text_model.encoder.layers.19.mlp.fc1.weight', 'text_model.encoder.layers.19.mlp.fc2.bias', 'text_model.encoder.layers.19.mlp.fc2.weight', 'text_model.encoder.layers.19.self_attn.k_proj.bias', 'text_model.encoder.layers.19.self_attn.k_proj.weight', 'text_model.encoder.layers.19.self_attn.out_proj.bias', 'text_model.encoder.layers.19.self_attn.out_proj.weight', 'text_model.encoder.layers.19.self_attn.q_proj.bias', 'text_model.encoder.layers.19.self_attn.q_proj.weight', 'text_model.encoder.layers.19.self_attn.v_proj.bias', 'text_model.encoder.layers.19.self_attn.v_proj.weight', 'text_model.encoder.layers.2.layer_norm1.bias', 'text_model.encoder.layers.2.layer_norm1.weight', 'text_model.encoder.layers.2.layer_norm2.bias', 'text_model.encoder.layers.2.layer_norm2.weight', 'text_model.encoder.layers.2.mlp.fc1.bias', 'text_model.encoder.layers.2.mlp.fc1.weight', 'text_model.encoder.layers.2.mlp.fc2.bias', 'text_model.encoder.layers.2.mlp.fc2.weight', 'text_model.encoder.layers.2.self_attn.k_proj.bias', 'text_model.encoder.layers.2.self_attn.k_proj.weight', 'text_model.encoder.layers.2.self_attn.out_proj.bias', 'text_model.encoder.layers.2.self_attn.out_proj.weight', 'text_model.encoder.layers.2.self_attn.q_proj.bias', 'text_model.encoder.layers.2.self_attn.q_proj.weight', 'text_model.encoder.layers.2.self_attn.v_proj.bias', 'text_model.encoder.layers.2.self_attn.v_proj.weight', 'text_model.encoder.layers.20.layer_norm1.bias', 'text_model.encoder.layers.20.layer_norm1.weight', 'text_model.encoder.layers.20.layer_norm2.bias', 'text_model.encoder.layers.20.layer_norm2.weight', 'text_model.encoder.layers.20.mlp.fc1.bias', 'text_model.encoder.layers.20.mlp.fc1.weight', 'text_model.encoder.layers.20.mlp.fc2.bias', 'text_model.encoder.layers.20.mlp.fc2.weight', 'text_model.encoder.layers.20.self_attn.k_proj.bias', 'text_model.encoder.layers.20.self_attn.k_proj.weight', 'text_model.encoder.layers.20.self_attn.out_proj.bias', 'text_model.encoder.layers.20.self_attn.out_proj.weight', 'text_model.encoder.layers.20.self_attn.q_proj.bias', 'text_model.encoder.layers.20.self_attn.q_proj.weight', 'text_model.encoder.layers.20.self_attn.v_proj.bias', 'text_model.encoder.layers.20.self_attn.v_proj.weight', 'text_model.encoder.layers.21.layer_norm1.bias', 'text_model.encoder.layers.21.layer_norm1.weight', 'text_model.encoder.layers.21.layer_norm2.bias', 'text_model.encoder.layers.21.layer_norm2.weight', 'text_model.encoder.layers.21.mlp.fc1.bias', 'text_model.encoder.layers.21.mlp.fc1.weight', 'text_model.encoder.layers.21.mlp.fc2.bias', 'text_model.encoder.layers.21.mlp.fc2.weight', 'text_model.encoder.layers.21.self_attn.k_proj.bias', 'text_model.encoder.layers.21.self_attn.k_proj.weight', 'text_model.encoder.layers.21.self_attn.out_proj.bias', 'text_model.encoder.layers.21.self_attn.out_proj.weight', 'text_model.encoder.layers.21.self_attn.q_proj.bias', 'text_model.encoder.layers.21.self_attn.q_proj.weight', 'text_model.encoder.layers.21.self_attn.v_proj.bias', 'text_model.encoder.layers.21.self_attn.v_proj.weight', 'text_model.encoder.layers.22.layer_norm1.bias', 'text_model.encoder.layers.22.layer_norm1.weight', 'text_model.encoder.layers.22.layer_norm2.bias', 'text_model.encoder.layers.22.layer_norm2.weight', 'text_model.encoder.layers.22.mlp.fc1.bias', 'text_model.encoder.layers.22.mlp.fc1.weight', 'text_model.encoder.layers.22.mlp.fc2.bias', 'text_model.encoder.layers.22.mlp.fc2.weight', 'text_model.encoder.layers.22.self_attn.k_proj.bias', 'text_model.encoder.layers.22.self_attn.k_proj.weight', 'text_model.encoder.layers.22.self_attn.out_proj.bias', 'text_model.encoder.layers.22.self_attn.out_proj.weight', 'text_model.encoder.layers.22.self_attn.q_proj.bias', 'text_model.encoder.layers.22.self_attn.q_proj.weight', 'text_model.encoder.layers.22.self_attn.v_proj.bias', 'text_model.encoder.layers.22.self_attn.v_proj.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.24.layer_norm1.bias', 'text_model.encoder.layers.24.layer_norm1.weight', 'text_model.encoder.layers.24.layer_norm2.bias', 'text_model.encoder.layers.24.layer_norm2.weight', 'text_model.encoder.layers.24.mlp.fc1.bias', 'text_model.encoder.layers.24.mlp.fc1.weight', 'text_model.encoder.layers.24.mlp.fc2.bias', 'text_model.encoder.layers.24.mlp.fc2.weight', 'text_model.encoder.layers.24.self_attn.k_proj.bias', 'text_model.encoder.layers.24.self_attn.k_proj.weight', 'text_model.encoder.layers.24.self_attn.out_proj.bias', 'text_model.encoder.layers.24.self_attn.out_proj.weight', 'text_model.encoder.layers.24.self_attn.q_proj.bias', 'text_model.encoder.layers.24.self_attn.q_proj.weight', 'text_model.encoder.layers.24.self_attn.v_proj.bias', 'text_model.encoder.layers.24.self_attn.v_proj.weight', 'text_model.encoder.layers.25.layer_norm1.bias', 'text_model.encoder.layers.25.layer_norm1.weight', 'text_model.encoder.layers.25.layer_norm2.bias', 'text_model.encoder.layers.25.layer_norm2.weight', 'text_model.encoder.layers.25.mlp.fc1.bias', 'text_model.encoder.layers.25.mlp.fc1.weight', 'text_model.encoder.layers.25.mlp.fc2.bias', 'text_model.encoder.layers.25.mlp.fc2.weight', 'text_model.encoder.layers.25.self_attn.k_proj.bias', 'text_model.encoder.layers.25.self_attn.k_proj.weight', 'text_model.encoder.layers.25.self_attn.out_proj.bias', 'text_model.encoder.layers.25.self_attn.out_proj.weight', 'text_model.encoder.layers.25.self_attn.q_proj.bias', 'text_model.encoder.layers.25.self_attn.q_proj.weight', 'text_model.encoder.layers.25.self_attn.v_proj.bias', 'text_model.encoder.layers.25.self_attn.v_proj.weight', 'text_model.encoder.layers.26.layer_norm1.bias', 'text_model.encoder.layers.26.layer_norm1.weight', 'text_model.encoder.layers.26.layer_norm2.bias', 'text_model.encoder.layers.26.layer_norm2.weight', 'text_model.encoder.layers.26.mlp.fc1.bias', 'text_model.encoder.layers.26.mlp.fc1.weight', 'text_model.encoder.layers.26.mlp.fc2.bias', 'text_model.encoder.layers.26.mlp.fc2.weight', 'text_model.encoder.layers.26.self_attn.k_proj.bias', 'text_model.encoder.layers.26.self_attn.k_proj.weight', 'text_model.encoder.layers.26.self_attn.out_proj.bias', 'text_model.encoder.layers.26.self_attn.out_proj.weight', 'text_model.encoder.layers.26.self_attn.q_proj.bias', 'text_model.encoder.layers.26.self_attn.q_proj.weight', 'text_model.encoder.layers.26.self_attn.v_proj.bias', 'text_model.encoder.layers.26.self_attn.v_proj.weight', 'text_model.encoder.layers.3.layer_norm1.bias', 'text_model.encoder.layers.3.layer_norm1.weight', 'text_model.encoder.layers.3.layer_norm2.bias', 'text_model.encoder.layers.3.layer_norm2.weight', 'text_model.encoder.layers.3.mlp.fc1.bias', 'text_model.encoder.layers.3.mlp.fc1.weight', 'text_model.encoder.layers.3.mlp.fc2.bias', 'text_model.encoder.layers.3.mlp.fc2.weight', 'text_model.encoder.layers.3.self_attn.k_proj.bias', 'text_model.encoder.layers.3.self_attn.k_proj.weight', 'text_model.encoder.layers.3.self_attn.out_proj.bias', 'text_model.encoder.layers.3.self_attn.out_proj.weight', 'text_model.encoder.layers.3.self_attn.q_proj.bias', 'text_model.encoder.layers.3.self_attn.q_proj.weight', 'text_model.encoder.layers.3.self_attn.v_proj.bias', 'text_model.encoder.layers.3.self_attn.v_proj.weight', 'text_model.encoder.layers.4.layer_norm1.bias', 'text_model.encoder.layers.4.layer_norm1.weight', 'text_model.encoder.layers.4.layer_norm2.bias', 'text_model.encoder.layers.4.layer_norm2.weight', 'text_model.encoder.layers.4.mlp.fc1.bias', 'text_model.encoder.layers.4.mlp.fc1.weight', 'text_model.encoder.layers.4.mlp.fc2.bias', 'text_model.encoder.layers.4.mlp.fc2.weight', 'text_model.encoder.layers.4.self_attn.k_proj.bias', 'text_model.encoder.layers.4.self_attn.k_proj.weight', 'text_model.encoder.layers.4.self_attn.out_proj.bias', 'text_model.encoder.layers.4.self_attn.out_proj.weight', 'text_model.encoder.layers.4.self_attn.q_proj.bias', 'text_model.encoder.layers.4.self_attn.q_proj.weight', 'text_model.encoder.layers.4.self_attn.v_proj.bias', 'text_model.encoder.layers.4.self_attn.v_proj.weight', 'text_model.encoder.layers.5.layer_norm1.bias', 'text_model.encoder.layers.5.layer_norm1.weight', 'text_model.encoder.layers.5.layer_norm2.bias', 'text_model.encoder.layers.5.layer_norm2.weight', 'text_model.encoder.layers.5.mlp.fc1.bias', 'text_model.encoder.layers.5.mlp.fc1.weight', 'text_model.encoder.layers.5.mlp.fc2.bias', 'text_model.encoder.layers.5.mlp.fc2.weight', 'text_model.encoder.layers.5.self_attn.k_proj.bias', 'text_model.encoder.layers.5.self_attn.k_proj.weight', 'text_model.encoder.layers.5.self_attn.out_proj.bias', 'text_model.encoder.layers.5.self_attn.out_proj.weight', 'text_model.encoder.layers.5.self_attn.q_proj.bias', 'text_model.encoder.layers.5.self_attn.q_proj.weight', 'text_model.encoder.layers.5.self_attn.v_proj.bias', 'text_model.encoder.layers.5.self_attn.v_proj.weight', 'text_model.encoder.layers.6.layer_norm1.bias', 'text_model.encoder.layers.6.layer_norm1.weight', 'text_model.encoder.layers.6.layer_norm2.bias', 'text_model.encoder.layers.6.layer_norm2.weight', 'text_model.encoder.layers.6.mlp.fc1.bias', 'text_model.encoder.layers.6.mlp.fc1.weight', 'text_model.encoder.layers.6.mlp.fc2.bias', 'text_model.encoder.layers.6.mlp.fc2.weight', 'text_model.encoder.layers.6.self_attn.k_proj.bias', 'text_model.encoder.layers.6.self_attn.k_proj.weight', 'text_model.encoder.layers.6.self_attn.out_proj.bias', 'text_model.encoder.layers.6.self_attn.out_proj.weight', 'text_model.encoder.layers.6.self_attn.q_proj.bias', 'text_model.encoder.layers.6.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.v_proj.bias', 'text_model.encoder.layers.6.self_attn.v_proj.weight', 'text_model.encoder.layers.7.layer_norm1.bias', 'text_model.encoder.layers.7.layer_norm1.weight', 'text_model.encoder.layers.7.layer_norm2.bias', 'text_model.encoder.layers.7.layer_norm2.weight', 'text_model.encoder.layers.7.mlp.fc1.bias', 'text_model.encoder.layers.7.mlp.fc1.weight', 'text_model.encoder.layers.7.mlp.fc2.bias', 'text_model.encoder.layers.7.mlp.fc2.weight', 'text_model.encoder.layers.7.self_attn.k_proj.bias', 'text_model.encoder.layers.7.self_attn.k_proj.weight', 'text_model.encoder.layers.7.self_attn.out_proj.bias', 'text_model.encoder.layers.7.self_attn.out_proj.weight', 'text_model.encoder.layers.7.self_attn.q_proj.bias', 'text_model.encoder.layers.7.self_attn.q_proj.weight', 'text_model.encoder.layers.7.self_attn.v_proj.bias', 'text_model.encoder.layers.7.self_attn.v_proj.weight', 'text_model.encoder.layers.8.layer_norm1.bias', 'text_model.encoder.layers.8.layer_norm1.weight', 'text_model.encoder.layers.8.layer_norm2.bias', 'text_model.encoder.layers.8.layer_norm2.weight', 'text_model.encoder.layers.8.mlp.fc1.bias', 'text_model.encoder.layers.8.mlp.fc1.weight', 'text_model.encoder.layers.8.mlp.fc2.bias', 'text_model.encoder.layers.8.mlp.fc2.weight', 'text_model.encoder.layers.8.self_attn.k_proj.bias', 'text_model.encoder.layers.8.self_attn.k_proj.weight', 'text_model.encoder.layers.8.self_attn.out_proj.bias', 'text_model.encoder.layers.8.self_attn.out_proj.weight', 'text_model.encoder.layers.8.self_attn.q_proj.bias', 'text_model.encoder.layers.8.self_attn.q_proj.weight', 'text_model.encoder.layers.8.self_attn.v_proj.bias', 'text_model.encoder.layers.8.self_attn.v_proj.weight', 'text_model.encoder.layers.9.layer_norm1.bias', 'text_model.encoder.layers.9.layer_norm1.weight', 'text_model.encoder.layers.9.layer_norm2.bias', 'text_model.encoder.layers.9.layer_norm2.weight', 'text_model.encoder.layers.9.mlp.fc1.bias', 'text_model.encoder.layers.9.mlp.fc1.weight', 'text_model.encoder.layers.9.mlp.fc2.bias', 'text_model.encoder.layers.9.mlp.fc2.weight', 'text_model.encoder.layers.9.self_attn.k_proj.bias', 'text_model.encoder.layers.9.self_attn.k_proj.weight', 'text_model.encoder.layers.9.self_attn.out_proj.bias', 'text_model.encoder.layers.9.self_attn.out_proj.weight', 'text_model.encoder.layers.9.self_attn.q_proj.bias', 'text_model.encoder.layers.9.self_attn.q_proj.weight', 'text_model.encoder.layers.9.self_attn.v_proj.bias', 'text_model.encoder.layers.9.self_attn.v_proj.weight', 'text_model.final_layer_norm.bias', 'text_model.final_layer_norm.weight', 'text_model.head.bias', 'text_model.head.weight'] - This IS expected if you are initializing SiglipVisionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing SiglipVisionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 2025-02-15 03:04:04,266 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of SiglipVisionModel were initialized from the model checkpoint at google/siglip-so400m-patch14-384. If your task is similar to the task the model of the checkpoint was trained on, you can already use SiglipVisionModel for predictions without further training. 2025-02-15 03:04:04,771 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/preprocessor_config.json 2025-02-15 03:04:04,772 - image_processing_base.py:429 - from_dict - INFO - Image processor SiglipImageProcessor { "do_convert_rgb": null, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.5, 0.5, 0.5 ], "image_processor_type": "SiglipImageProcessor", "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "SiglipProcessor", "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "height": 384, "width": 384 } } 2025-02-15 03:04:05,149 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-15 03:04:05,153 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-15 03:04:05,153 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/model.safetensors 2025-02-15 03:04:05,808 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing Dinov2Model. 2025-02-15 03:04:05,809 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of Dinov2Model were initialized from the model checkpoint at facebook/dinov2-giant. If your task is similar to the task the model of the checkpoint was trained on, you can already use Dinov2Model for predictions without further training. 2025-02-15 03:04:06,009 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/preprocessor_config.json 2025-02-15 03:04:06,011 - image_processing_base.py:429 - from_dict - INFO - Image processor BitImageProcessor { "crop_size": { "height": 378, "width": 378 }, "do_center_crop": true, "do_convert_rgb": true, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.485, 0.456, 0.406 ], "image_processor_type": "BitImageProcessor", "image_std": [ 0.229, 0.224, 0.225 ], "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "shortest_edge": 378 } } 2025-02-15 03:04:06,718 - finetune_llama.py:1239 - train - INFO - Total params: 3264865280 2025-02-15 03:04:06,718 - finetune_llama.py:1240 - train - INFO - Trainable params: 12589056 2025-02-15 03:04:06,718 - finetune_llama.py:1241 - train - INFO - LM head params: 394002432 2025-02-15 03:04:09,067 - trainer_callback.py:423 - add_callback - WARNING - You are adding a to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is :DefaultFlowCallback TensorBoardCallback 2025-02-15 03:04:09,067 - trainer.py:648 - __init__ - INFO - Using auto half precision backend 2025-02-15 03:04:09,383 - trainer.py:2134 - _inner_training_loop - INFO - ***** Running training ***** 2025-02-15 03:04:09,383 - trainer.py:2135 - _inner_training_loop - INFO - Num examples = 540 2025-02-15 03:04:09,383 - trainer.py:2136 - _inner_training_loop - INFO - Num Epochs = 2 2025-02-15 03:04:09,383 - trainer.py:2137 - _inner_training_loop - INFO - Instantaneous batch size per device = 1 2025-02-15 03:04:09,383 - trainer.py:2140 - _inner_training_loop - INFO - Total train batch size (w. parallel, distributed & accumulation) = 1 2025-02-15 03:04:09,383 - trainer.py:2141 - _inner_training_loop - INFO - Gradient Accumulation steps = 1 2025-02-15 03:04:09,383 - trainer.py:2142 - _inner_training_loop - INFO - Total optimization steps = 1,080 2025-02-15 03:04:09,384 - trainer.py:2143 - _inner_training_loop - INFO - Number of trainable parameters = 406,591,488 2025-02-15 03:05:13,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:13,844 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:05:13,898 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:05:13,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:13,908 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1244, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:05:13,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:13,911 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1244, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:05:33,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:05:33,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:05:33,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.23 seconds 2025-02-15 03:05:33,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:33,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19945.19 MB 2025-02-15 03:05:33,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24381.13 MB 2025-02-15 03:05:33,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4435.94 MB 2025-02-15 03:05:33,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21149.78 MB 2025-02-15 03:05:33,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25249.71 MB 2025-02-15 03:05:33,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4099.93 MB 2025-02-15 03:05:33,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33301.23 MB 2025-02-15 03:05:33,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:05:33,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:05:33,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 03:05:33,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:33,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24381.13 MB 2025-02-15 03:05:33,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20586.93 MB 2025-02-15 03:05:33,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3794.20 MB 2025-02-15 03:05:33,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25249.71 MB 2025-02-15 03:05:33,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36014.39 MB 2025-02-15 03:05:33,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10764.68 MB 2025-02-15 03:05:33,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37374.40 MB 2025-02-15 03:05:35,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:05:35,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:05:35,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 03:05:35,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20586.93 MB 2025-02-15 03:05:35,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21117.77 MB 2025-02-15 03:05:35,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:05:35,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36014.39 MB 2025-02-15 03:05:35,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22944.94 MB 2025-02-15 03:05:35,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13069.45 MB 2025-02-15 03:05:35,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25097.36 MB 2025-02-15 03:05:35,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:05:35,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:05:35,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:05:35,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21117.77 MB 2025-02-15 03:05:35,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.14 MB 2025-02-15 03:05:35,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.36 MB 2025-02-15 03:05:35,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22944.94 MB 2025-02-15 03:05:35,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25778.19 MB 2025-02-15 03:05:35,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 03:05:35,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24424.57 MB 2025-02-15 03:05:35,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:05:35,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:05:35,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:05:35,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.14 MB 2025-02-15 03:05:35,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25248.99 MB 2025-02-15 03:05:35,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:05:35,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25778.19 MB 2025-02-15 03:05:35,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32388.42 MB 2025-02-15 03:05:35,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6610.22 MB 2025-02-15 03:05:35,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30795.37 MB 2025-02-15 03:05:35,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:05:35,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:05:35,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:05:35,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21117.77 MB 2025-02-15 03:05:35,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25248.99 MB 2025-02-15 03:05:35,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.22 MB 2025-02-15 03:05:35,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22944.94 MB 2025-02-15 03:05:35,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32388.42 MB 2025-02-15 03:05:35,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9443.48 MB 2025-02-15 03:05:35,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30795.37 MB 2025-02-15 03:05:35,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:05:35,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:05:35,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:05:35,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26782.54 MB 2025-02-15 03:05:35,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27550.59 MB 2025-02-15 03:05:35,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 768.05 MB 2025-02-15 03:05:35,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32388.42 MB 2025-02-15 03:05:35,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32805.75 MB 2025-02-15 03:05:35,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:05:35,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28258.38 MB 2025-02-15 03:05:35,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:05:35,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:05:35,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 03:05:35,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27963.48 MB 2025-02-15 03:05:35,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28192.24 MB 2025-02-15 03:05:35,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.77 MB 2025-02-15 03:05:35,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32805.75 MB 2025-02-15 03:05:35,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32805.75 MB 2025-02-15 03:05:35,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:35,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28428.20 MB 2025-02-15 03:05:35,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:05:35,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:05:35,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.92 seconds 2025-02-15 03:05:35,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:35,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15610.21 MB 2025-02-15 03:05:35,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28393.10 MB 2025-02-15 03:05:35,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12782.88 MB 2025-02-15 03:05:35,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16812.87 MB 2025-02-15 03:05:35,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32805.75 MB 2025-02-15 03:05:35,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15992.88 MB 2025-02-15 03:05:35,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28428.20 MB 2025-02-15 03:05:35,863 - logging.py:328 - warning_once - WARNING - The attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.45 `position_ids` will be removed and `position_embeddings` will be mandatory. 2025-02-15 03:05:36,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:05:36,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:05:36,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 03:05:36,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:36,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17634.89 MB 2025-02-15 03:05:36,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20645.61 MB 2025-02-15 03:05:36,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.72 MB 2025-02-15 03:05:36,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32805.75 MB 2025-02-15 03:05:36,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32805.75 MB 2025-02-15 03:05:36,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:36,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20946.64 MB 2025-02-15 03:05:36,149 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 03:05:36,153 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:05:36,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:05:36,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:05:36,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 03:05:36,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:36,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20645.61 MB 2025-02-15 03:05:36,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29076.01 MB 2025-02-15 03:05:36,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 03:05:36,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32805.75 MB 2025-02-15 03:05:36,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41185.97 MB 2025-02-15 03:05:36,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 03:05:36,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29076.01 MB 2025-02-15 03:05:36,319 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 03:05:36,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:36,320 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:05:36,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:36,321 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:05:36,326 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:05:36,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:36,327 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:05:36,327 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:05:48,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:48,350 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:05:48,355 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:05:48,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:48,359 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:05:48,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:48,360 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:05:50,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:05:50,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:05:50,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.10 seconds 2025-02-15 03:05:50,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:50,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13916.38 MB 2025-02-15 03:05:50,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14398.72 MB 2025-02-15 03:05:50,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 482.34 MB 2025-02-15 03:05:50,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49566.19 MB 2025-02-15 03:05:50,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19314.77 MB 2025-02-15 03:05:50,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30251.42 MB 2025-02-15 03:05:50,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23387.75 MB 2025-02-15 03:05:50,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:05:50,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:05:50,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:05:50,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:50,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14398.72 MB 2025-02-15 03:05:50,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14588.72 MB 2025-02-15 03:05:50,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.00 MB 2025-02-15 03:05:50,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19314.77 MB 2025-02-15 03:05:50,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19314.77 MB 2025-02-15 03:05:50,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:50,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16224.77 MB 2025-02-15 03:05:51,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:05:51,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:05:51,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-15 03:05:51,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14588.72 MB 2025-02-15 03:05:51,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.18 MB 2025-02-15 03:05:51,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.46 MB 2025-02-15 03:05:51,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19314.77 MB 2025-02-15 03:05:51,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18842.91 MB 2025-02-15 03:05:51,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 03:05:51,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18759.34 MB 2025-02-15 03:05:51,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:05:51,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:05:51,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:05:51,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-15 03:05:51,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15375.13 MB 2025-02-15 03:05:51,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-15 03:05:51,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18842.91 MB 2025-02-15 03:05:51,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18842.91 MB 2025-02-15 03:05:51,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:51,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15835.80 MB 2025-02-15 03:05:51,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:05:51,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:05:51,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:05:51,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15375.13 MB 2025-02-15 03:05:51,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-15 03:05:51,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-15 03:05:51,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18842.91 MB 2025-02-15 03:05:51,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18842.91 MB 2025-02-15 03:05:51,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:51,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-15 03:05:51,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:05:51,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:05:51,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:05:51,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-15 03:05:51,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-15 03:05:51,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-15 03:05:51,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18842.91 MB 2025-02-15 03:05:51,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18842.91 MB 2025-02-15 03:05:51,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:05:51,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-15 03:05:51,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:05:51,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:05:51,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:05:51,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16602.18 MB 2025-02-15 03:05:51,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16851.45 MB 2025-02-15 03:05:51,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-15 03:05:51,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18842.91 MB 2025-02-15 03:05:51,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18972.93 MB 2025-02-15 03:05:51,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-15 03:05:51,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17093.03 MB 2025-02-15 03:05:51,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:05:51,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:05:51,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:05:51,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16985.65 MB 2025-02-15 03:05:51,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17190.69 MB 2025-02-15 03:05:51,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.04 MB 2025-02-15 03:05:51,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18972.93 MB 2025-02-15 03:05:51,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18977.13 MB 2025-02-15 03:05:51,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 03:05:51,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17204.15 MB 2025-02-15 03:05:51,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:05:51,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:05:51,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-15 03:05:51,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13442.54 MB 2025-02-15 03:05:51,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.39 MB 2025-02-15 03:05:51,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3948.85 MB 2025-02-15 03:05:51,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49566.19 MB 2025-02-15 03:05:51,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18977.13 MB 2025-02-15 03:05:51,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30589.06 MB 2025-02-15 03:05:51,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17391.39 MB 2025-02-15 03:05:51,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:05:51,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:05:51,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:05:51,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.39 MB 2025-02-15 03:05:51,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20399.90 MB 2025-02-15 03:05:51,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-15 03:05:51,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18977.13 MB 2025-02-15 03:05:51,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22064.14 MB 2025-02-15 03:05:51,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3087.01 MB 2025-02-15 03:05:51,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20701.22 MB 2025-02-15 03:05:51,527 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 03:05:51,528 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:05:51,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:05:51,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:05:51,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:05:51,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:05:51,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20399.90 MB 2025-02-15 03:05:51,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28823.10 MB 2025-02-15 03:05:51,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 03:05:51,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22064.14 MB 2025-02-15 03:05:51,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32535.22 MB 2025-02-15 03:05:51,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 03:05:51,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28823.10 MB 2025-02-15 03:05:51,691 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 03:05:51,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:51,692 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:05:51,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:51,693 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:05:51,698 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:05:51,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:05:51,699 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:05:51,699 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:06:48,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:48,746 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:06:48,752 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:06:48,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:48,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 246, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:06:48,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:48,757 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 246, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:06:52,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:06:52,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:06:52,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.76 seconds 2025-02-15 03:06:52,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:52,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20015.33 MB 2025-02-15 03:06:52,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20885.91 MB 2025-02-15 03:06:52,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 870.58 MB 2025-02-15 03:06:52,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40911.24 MB 2025-02-15 03:06:52,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24175.97 MB 2025-02-15 03:06:52,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16735.27 MB 2025-02-15 03:06:52,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29714.00 MB 2025-02-15 03:06:52,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:06:52,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:06:52,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:06:52,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:52,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20885.91 MB 2025-02-15 03:06:52,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20162.96 MB 2025-02-15 03:06:52,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -722.96 MB 2025-02-15 03:06:52,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 03:06:52,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24175.97 MB 2025-02-15 03:06:52,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:52,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22051.80 MB 2025-02-15 03:06:52,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:06:52,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:06:52,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-15 03:06:52,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:52,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20162.96 MB 2025-02-15 03:06:52,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20273.11 MB 2025-02-15 03:06:52,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 110.15 MB 2025-02-15 03:06:52,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 03:06:52,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22793.95 MB 2025-02-15 03:06:52,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1382.02 MB 2025-02-15 03:06:52,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24248.71 MB 2025-02-15 03:06:52,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:06:52,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:06:52,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:06:52,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:52,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20273.04 MB 2025-02-15 03:06:52,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20665.02 MB 2025-02-15 03:06:52,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 391.98 MB 2025-02-15 03:06:52,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 03:06:52,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22793.95 MB 2025-02-15 03:06:52,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:52,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20959.15 MB 2025-02-15 03:06:53,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:06:53,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:06:53,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:06:53,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20665.02 MB 2025-02-15 03:06:53,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21141.43 MB 2025-02-15 03:06:53,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 476.41 MB 2025-02-15 03:06:53,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 03:06:53,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22793.95 MB 2025-02-15 03:06:53,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:53,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22281.62 MB 2025-02-15 03:06:53,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:06:53,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:06:53,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:06:53,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20273.04 MB 2025-02-15 03:06:53,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21141.43 MB 2025-02-15 03:06:53,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 868.39 MB 2025-02-15 03:06:53,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 03:06:53,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22793.95 MB 2025-02-15 03:06:53,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:53,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22281.62 MB 2025-02-15 03:06:53,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:06:53,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:06:53,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:06:53,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21601.10 MB 2025-02-15 03:06:53,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16468.59 MB 2025-02-15 03:06:53,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5132.51 MB 2025-02-15 03:06:53,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 03:06:53,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22917.68 MB 2025-02-15 03:06:53,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 123.73 MB 2025-02-15 03:06:53,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21608.88 MB 2025-02-15 03:06:53,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:06:53,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:06:53,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 03:06:53,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16595.07 MB 2025-02-15 03:06:53,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16794.67 MB 2025-02-15 03:06:53,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.60 MB 2025-02-15 03:06:53,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22917.68 MB 2025-02-15 03:06:53,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22917.68 MB 2025-02-15 03:06:53,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:53,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16794.67 MB 2025-02-15 03:06:53,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:06:53,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:06:53,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.43 seconds 2025-02-15 03:06:53,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19158.25 MB 2025-02-15 03:06:53,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16971.96 MB 2025-02-15 03:06:53,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2186.29 MB 2025-02-15 03:06:53,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40911.24 MB 2025-02-15 03:06:53,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22917.68 MB 2025-02-15 03:06:53,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17993.56 MB 2025-02-15 03:06:53,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16971.96 MB 2025-02-15 03:06:53,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:06:53,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:06:53,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 03:06:53,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14308.22 MB 2025-02-15 03:06:53,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16965.77 MB 2025-02-15 03:06:53,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2657.55 MB 2025-02-15 03:06:53,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22917.68 MB 2025-02-15 03:06:53,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22917.68 MB 2025-02-15 03:06:53,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:06:53,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17231.50 MB 2025-02-15 03:06:53,467 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7195, cut from 7197 2025-02-15 03:06:53,468 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:06:53,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:06:53,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:06:53,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:06:53,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:06:53,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16965.77 MB 2025-02-15 03:06:53,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24407.32 MB 2025-02-15 03:06:53,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7441.55 MB 2025-02-15 03:06:53,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22917.68 MB 2025-02-15 03:06:53,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26617.05 MB 2025-02-15 03:06:53,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3699.38 MB 2025-02-15 03:06:53,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24407.32 MB 2025-02-15 03:06:53,696 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6987] 2025-02-15 03:06:53,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:53,699 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:06:53,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:53,701 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:06:53,708 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:06:53,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:06:53,710 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:06:53,711 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:07:00,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:00,629 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:07:00,635 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:07:00,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:00,638 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:07:00,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:00,639 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:07:21,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:07:21,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:07:21,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.90 seconds 2025-02-15 03:07:21,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:21,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-15 03:07:21,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-15 03:07:21,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-15 03:07:21,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34015.81 MB 2025-02-15 03:07:21,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36612.08 MB 2025-02-15 03:07:21,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2596.27 MB 2025-02-15 03:07:21,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-15 03:07:21,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:07:21,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:07:21,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:07:21,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:21,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-15 03:07:21,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-15 03:07:21,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-15 03:07:21,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36612.08 MB 2025-02-15 03:07:21,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45069.89 MB 2025-02-15 03:07:21,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8457.81 MB 2025-02-15 03:07:21,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40081.89 MB 2025-02-15 03:07:23,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:07:23,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:07:23,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 03:07:23,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-15 03:07:23,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-15 03:07:23,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:07:23,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45069.89 MB 2025-02-15 03:07:23,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31763.46 MB 2025-02-15 03:07:23,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13306.43 MB 2025-02-15 03:07:23,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27409.44 MB 2025-02-15 03:07:23,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:07:23,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:07:23,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:07:23,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-15 03:07:23,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-15 03:07:23,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:07:23,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31763.46 MB 2025-02-15 03:07:23,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31763.46 MB 2025-02-15 03:07:23,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:07:23,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-15 03:07:23,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:07:23,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:07:23,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:07:23,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-15 03:07:23,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-15 03:07:23,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:07:23,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31763.46 MB 2025-02-15 03:07:23,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35538.34 MB 2025-02-15 03:07:23,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:07:23,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-15 03:07:23,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:07:23,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:07:23,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:07:23,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-15 03:07:23,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-15 03:07:23,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:07:23,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31763.46 MB 2025-02-15 03:07:23,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35538.34 MB 2025-02-15 03:07:23,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:07:23,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-15 03:07:23,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:07:23,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:07:23,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:07:23,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-15 03:07:23,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29862.83 MB 2025-02-15 03:07:23,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:07:23,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35538.34 MB 2025-02-15 03:07:23,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-15 03:07:23,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:07:23,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.62 MB 2025-02-15 03:07:23,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:07:23,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:07:23,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:07:23,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30275.72 MB 2025-02-15 03:07:23,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30503.46 MB 2025-02-15 03:07:23,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.74 MB 2025-02-15 03:07:23,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-15 03:07:23,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-15 03:07:23,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:07:23,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30718.98 MB 2025-02-15 03:07:23,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:07:23,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:07:23,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.30 seconds 2025-02-15 03:07:23,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:23,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-15 03:07:23,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30704.31 MB 2025-02-15 03:07:23,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12962.42 MB 2025-02-15 03:07:23,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34015.81 MB 2025-02-15 03:07:23,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-15 03:07:23,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1939.87 MB 2025-02-15 03:07:23,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30718.98 MB 2025-02-15 03:07:24,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:07:24,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:07:24,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:07:24,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:24,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30704.31 MB 2025-02-15 03:07:24,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22736.09 MB 2025-02-15 03:07:24,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7968.23 MB 2025-02-15 03:07:24,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-15 03:07:24,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-15 03:07:24,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:07:24,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33207.38 MB 2025-02-15 03:07:24,223 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-15 03:07:24,223 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:07:24,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:07:24,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:07:24,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:07:24,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:07:24,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22736.09 MB 2025-02-15 03:07:24,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31145.89 MB 2025-02-15 03:07:24,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-15 03:07:24,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-15 03:07:24,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40135.29 MB 2025-02-15 03:07:24,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 03:07:24,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31145.89 MB 2025-02-15 03:07:24,386 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-15 03:07:24,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:24,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:07:24,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:24,389 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:07:24,393 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:07:24,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:07:24,394 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:07:24,394 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:09:31,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:31,698 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:09:31,704 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:09:31,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:31,709 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 111, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:09:31,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:31,710 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 111, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:09:33,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:09:33,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:09:33,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.72 seconds 2025-02-15 03:09:33,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:33,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13742.17 MB 2025-02-15 03:09:33,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14134.99 MB 2025-02-15 03:09:33,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.82 MB 2025-02-15 03:09:33,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52676.26 MB 2025-02-15 03:09:33,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 03:09:33,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35253.13 MB 2025-02-15 03:09:33,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22987.05 MB 2025-02-15 03:09:33,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:09:33,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:09:33,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:09:33,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:33,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14134.99 MB 2025-02-15 03:09:33,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14325.32 MB 2025-02-15 03:09:33,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.32 MB 2025-02-15 03:09:33,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 03:09:33,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 03:09:33,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:09:33,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14914.61 MB 2025-02-15 03:09:33,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:09:33,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:09:33,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-15 03:09:33,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:33,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14325.32 MB 2025-02-15 03:09:33,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14472.63 MB 2025-02-15 03:09:33,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.31 MB 2025-02-15 03:09:33,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 03:09:33,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 03:09:33,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:09:33,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18411.07 MB 2025-02-15 03:09:33,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:09:33,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:09:33,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:09:33,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:33,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14472.56 MB 2025-02-15 03:09:33,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.78 MB 2025-02-15 03:09:33,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 524.22 MB 2025-02-15 03:09:33,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 03:09:33,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 03:09:33,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:09:33,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15390.12 MB 2025-02-15 03:09:34,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:09:34,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:09:34,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:09:34,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.78 MB 2025-02-15 03:09:34,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15633.97 MB 2025-02-15 03:09:34,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.20 MB 2025-02-15 03:09:34,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 03:09:34,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17685.28 MB 2025-02-15 03:09:34,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 262.14 MB 2025-02-15 03:09:34,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17158.18 MB 2025-02-15 03:09:34,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:09:34,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:09:34,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 03:09:34,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14472.56 MB 2025-02-15 03:09:34,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15633.97 MB 2025-02-15 03:09:34,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1161.42 MB 2025-02-15 03:09:34,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 03:09:34,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17685.28 MB 2025-02-15 03:09:34,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 262.14 MB 2025-02-15 03:09:34,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17158.18 MB 2025-02-15 03:09:34,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:09:34,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:09:34,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:09:34,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16248.67 MB 2025-02-15 03:09:34,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16516.07 MB 2025-02-15 03:09:34,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.40 MB 2025-02-15 03:09:34,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17685.28 MB 2025-02-15 03:09:34,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-15 03:09:34,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 03:09:34,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16712.48 MB 2025-02-15 03:09:34,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:09:34,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:09:34,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:09:34,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16685.22 MB 2025-02-15 03:09:34,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16914.23 MB 2025-02-15 03:09:34,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-15 03:09:34,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-15 03:09:34,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-15 03:09:34,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:09:34,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16914.23 MB 2025-02-15 03:09:34,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:09:34,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:09:34,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-15 03:09:34,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13355.44 MB 2025-02-15 03:09:34,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17114.74 MB 2025-02-15 03:09:34,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3759.30 MB 2025-02-15 03:09:34,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52676.26 MB 2025-02-15 03:09:34,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-15 03:09:34,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34821.11 MB 2025-02-15 03:09:34,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17114.74 MB 2025-02-15 03:09:34,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:09:34,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:09:34,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:09:34,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17114.74 MB 2025-02-15 03:09:34,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20120.29 MB 2025-02-15 03:09:34,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-15 03:09:34,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-15 03:09:34,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21747.47 MB 2025-02-15 03:09:34,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3892.31 MB 2025-02-15 03:09:34,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20421.45 MB 2025-02-15 03:09:34,448 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 03:09:34,449 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:09:34,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:09:34,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:09:34,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:09:34,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:09:34,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.29 MB 2025-02-15 03:09:34,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28535.24 MB 2025-02-15 03:09:34,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 03:09:34,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21747.47 MB 2025-02-15 03:09:34,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32208.06 MB 2025-02-15 03:09:34,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-15 03:09:34,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28535.24 MB 2025-02-15 03:09:34,612 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 03:09:34,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:34,613 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:09:34,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:34,615 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:09:34,620 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:09:34,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:09:34,621 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:09:34,621 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:10:54,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:10:54,132 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:10:54,137 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:10:54,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:10:54,141 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2959, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:10:54,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:10:54,142 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2959, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:11:39,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:11:39,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:11:39,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.57 seconds 2025-02-15 03:11:39,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:39,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38779.55 MB 2025-02-15 03:11:39,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49251.29 MB 2025-02-15 03:11:39,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10471.74 MB 2025-02-15 03:11:39,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61199.09 MB 2025-02-15 03:11:39,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51764.00 MB 2025-02-15 03:11:39,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9435.09 MB 2025-02-15 03:11:39,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59723.02 MB 2025-02-15 03:11:40,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:11:40,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:11:40,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-15 03:11:40,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:40,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49251.29 MB 2025-02-15 03:11:40,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36353.15 MB 2025-02-15 03:11:40,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12898.13 MB 2025-02-15 03:11:40,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51764.00 MB 2025-02-15 03:11:40,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74371.30 MB 2025-02-15 03:11:40,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22607.30 MB 2025-02-15 03:11:40,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 80052.41 MB 2025-02-15 03:11:41,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:11:41,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:11:41,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:11:41,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:41,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36353.15 MB 2025-02-15 03:11:41,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36884.00 MB 2025-02-15 03:11:41,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:11:41,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74371.30 MB 2025-02-15 03:11:41,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38711.33 MB 2025-02-15 03:11:41,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35659.97 MB 2025-02-15 03:11:41,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40862.54 MB 2025-02-15 03:11:42,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:11:42,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:11:42,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:11:42,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36884.00 MB 2025-02-15 03:11:42,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38773.27 MB 2025-02-15 03:11:42,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 03:11:42,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38711.33 MB 2025-02-15 03:11:42,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42486.20 MB 2025-02-15 03:11:42,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:11:42,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40190.70 MB 2025-02-15 03:11:42,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:11:42,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:11:42,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:11:42,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38773.27 MB 2025-02-15 03:11:42,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41015.12 MB 2025-02-15 03:11:42,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:11:42,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42486.20 MB 2025-02-15 03:11:42,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48620.37 MB 2025-02-15 03:11:42,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:11:42,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46559.41 MB 2025-02-15 03:11:42,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:11:42,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:11:42,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 03:11:42,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36884.00 MB 2025-02-15 03:11:42,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41015.12 MB 2025-02-15 03:11:42,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 03:11:42,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38711.33 MB 2025-02-15 03:11:42,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48620.37 MB 2025-02-15 03:11:42,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 03:11:42,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46559.41 MB 2025-02-15 03:11:42,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:11:42,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:11:42,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:11:42,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42548.67 MB 2025-02-15 03:11:42,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43315.67 MB 2025-02-15 03:11:42,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:11:42,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48620.37 MB 2025-02-15 03:11:42,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49033.51 MB 2025-02-15 03:11:42,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:11:42,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44023.46 MB 2025-02-15 03:11:42,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:11:42,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:11:42,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:11:42,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43728.56 MB 2025-02-15 03:11:42,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43956.32 MB 2025-02-15 03:11:42,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.76 MB 2025-02-15 03:11:42,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49033.51 MB 2025-02-15 03:11:42,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49033.51 MB 2025-02-15 03:11:42,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:11:42,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44175.21 MB 2025-02-15 03:11:42,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:11:42,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:11:42,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 48.27 seconds 2025-02-15 03:11:42,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28469.44 MB 2025-02-15 03:11:42,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44157.17 MB 2025-02-15 03:11:42,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15687.73 MB 2025-02-15 03:11:42,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50887.39 MB 2025-02-15 03:11:42,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49033.51 MB 2025-02-15 03:11:42,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1853.88 MB 2025-02-15 03:11:42,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44175.21 MB 2025-02-15 03:11:42,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:11:42,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:11:42,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:11:42,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44157.17 MB 2025-02-15 03:11:42,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33460.42 MB 2025-02-15 03:11:42,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10696.75 MB 2025-02-15 03:11:42,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49033.51 MB 2025-02-15 03:11:42,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49033.51 MB 2025-02-15 03:11:42,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:11:42,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46657.77 MB 2025-02-15 03:11:42,701 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-15 03:11:42,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:11:42,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:11:42,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:11:42,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:11:42,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:11:42,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33460.42 MB 2025-02-15 03:11:42,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41861.95 MB 2025-02-15 03:11:42,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-15 03:11:42,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49033.51 MB 2025-02-15 03:11:42,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53211.04 MB 2025-02-15 03:11:42,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 03:11:42,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41861.95 MB 2025-02-15 03:11:42,865 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-15 03:11:42,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:11:42,866 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:11:42,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:11:42,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:11:42,872 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:11:42,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:11:42,873 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:11:42,873 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:13:18,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:18,386 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:13:18,394 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:13:18,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:18,401 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1655, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:13:18,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:18,403 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1655, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:13:43,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:13:43,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:13:43,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.38 seconds 2025-02-15 03:13:43,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:43,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29691.63 MB 2025-02-15 03:13:43,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35548.97 MB 2025-02-15 03:13:43,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5857.35 MB 2025-02-15 03:13:43,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61566.09 MB 2025-02-15 03:13:43,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-15 03:13:43,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17431.53 MB 2025-02-15 03:13:43,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44372.32 MB 2025-02-15 03:13:43,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:13:43,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:13:43,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:13:43,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:43,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35548.97 MB 2025-02-15 03:13:43,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29572.30 MB 2025-02-15 03:13:43,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5976.68 MB 2025-02-15 03:13:43,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-15 03:13:43,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57078.19 MB 2025-02-15 03:13:43,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12943.62 MB 2025-02-15 03:13:43,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52883.82 MB 2025-02-15 03:13:45,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:13:45,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:13:45,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:13:45,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:45,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29572.30 MB 2025-02-15 03:13:45,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30103.14 MB 2025-02-15 03:13:45,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:13:45,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57078.19 MB 2025-02-15 03:13:45,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35515.27 MB 2025-02-15 03:13:45,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21562.92 MB 2025-02-15 03:13:45,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34081.68 MB 2025-02-15 03:13:45,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:13:45,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:13:45,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:13:45,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:45,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30103.14 MB 2025-02-15 03:13:45,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31992.67 MB 2025-02-15 03:13:45,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:13:45,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35515.27 MB 2025-02-15 03:13:45,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36458.99 MB 2025-02-15 03:13:45,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:13:45,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33410.10 MB 2025-02-15 03:13:46,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:13:46,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:13:46,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:13:46,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31992.67 MB 2025-02-15 03:13:46,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34234.53 MB 2025-02-15 03:13:46,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:13:46,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36458.99 MB 2025-02-15 03:13:46,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42593.16 MB 2025-02-15 03:13:46,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:13:46,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39778.81 MB 2025-02-15 03:13:46,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:13:46,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:13:46,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:13:46,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30103.14 MB 2025-02-15 03:13:46,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34234.53 MB 2025-02-15 03:13:46,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:13:46,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35515.27 MB 2025-02-15 03:13:46,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42593.16 MB 2025-02-15 03:13:46,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 03:13:46,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39778.81 MB 2025-02-15 03:13:46,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:13:46,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:13:46,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:13:46,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35768.07 MB 2025-02-15 03:13:46,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36535.07 MB 2025-02-15 03:13:46,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:13:46,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42593.16 MB 2025-02-15 03:13:46,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-15 03:13:46,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 03:13:46,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37242.86 MB 2025-02-15 03:13:46,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:13:46,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:13:46,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:13:46,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36947.96 MB 2025-02-15 03:13:46,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37176.90 MB 2025-02-15 03:13:46,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-15 03:13:46,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43004.20 MB 2025-02-15 03:13:46,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-15 03:13:46,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:13:46,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37423.37 MB 2025-02-15 03:13:46,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:13:46,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:13:46,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.82 seconds 2025-02-15 03:13:46,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23925.48 MB 2025-02-15 03:13:46,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37377.75 MB 2025-02-15 03:13:46,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13452.27 MB 2025-02-15 03:13:46,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61566.09 MB 2025-02-15 03:13:46,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-15 03:13:46,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18561.89 MB 2025-02-15 03:13:46,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37423.37 MB 2025-02-15 03:13:46,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:13:46,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:13:46,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:13:46,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37377.75 MB 2025-02-15 03:13:46,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28926.44 MB 2025-02-15 03:13:46,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8451.31 MB 2025-02-15 03:13:46,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43004.20 MB 2025-02-15 03:13:46,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-15 03:13:46,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:13:46,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39886.65 MB 2025-02-15 03:13:46,516 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 03:13:46,516 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:13:46,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:13:46,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:13:46,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:13:46,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:13:46,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28926.44 MB 2025-02-15 03:13:46,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37356.84 MB 2025-02-15 03:13:46,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 03:13:46,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43004.20 MB 2025-02-15 03:13:46,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51384.42 MB 2025-02-15 03:13:46,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 03:13:46,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37356.84 MB 2025-02-15 03:13:46,680 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 03:13:46,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:46,682 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:13:46,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:46,683 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:13:46,687 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:13:46,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:46,688 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:13:46,688 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:13:59,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:59,456 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:13:59,464 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:13:59,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:59,471 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2077, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:13:59,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:13:59,473 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2077, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:14:31,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:14:31,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:14:31,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.42 seconds 2025-02-15 03:14:31,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:31,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32632.19 MB 2025-02-15 03:14:31,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39982.71 MB 2025-02-15 03:14:31,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7350.52 MB 2025-02-15 03:14:31,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59764.64 MB 2025-02-15 03:14:31,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45642.42 MB 2025-02-15 03:14:31,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14122.22 MB 2025-02-15 03:14:31,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48899.14 MB 2025-02-15 03:14:32,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:14:32,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:14:32,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 03:14:32,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:32,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39982.71 MB 2025-02-15 03:14:32,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31766.14 MB 2025-02-15 03:14:32,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8216.57 MB 2025-02-15 03:14:32,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45642.42 MB 2025-02-15 03:14:32,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61064.87 MB 2025-02-15 03:14:32,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15422.46 MB 2025-02-15 03:14:32,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61044.52 MB 2025-02-15 03:14:34,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:14:34,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:14:34,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:14:34,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31766.14 MB 2025-02-15 03:14:34,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32296.98 MB 2025-02-15 03:14:34,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:14:34,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61064.87 MB 2025-02-15 03:14:34,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35529.95 MB 2025-02-15 03:14:34,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25534.92 MB 2025-02-15 03:14:34,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36275.53 MB 2025-02-15 03:14:34,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:14:34,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:14:34,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:14:34,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32296.98 MB 2025-02-15 03:14:34,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34186.52 MB 2025-02-15 03:14:34,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:14:34,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35529.95 MB 2025-02-15 03:14:34,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38361.10 MB 2025-02-15 03:14:34,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:14:34,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35603.95 MB 2025-02-15 03:14:34,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:14:34,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:14:34,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:14:34,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34186.52 MB 2025-02-15 03:14:34,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36428.37 MB 2025-02-15 03:14:34,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:14:34,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38361.10 MB 2025-02-15 03:14:34,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44495.27 MB 2025-02-15 03:14:34,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:14:34,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41972.66 MB 2025-02-15 03:14:34,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:14:34,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:14:34,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:14:34,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32296.98 MB 2025-02-15 03:14:34,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36428.37 MB 2025-02-15 03:14:34,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:14:34,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35529.95 MB 2025-02-15 03:14:34,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44495.27 MB 2025-02-15 03:14:34,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 03:14:34,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41972.66 MB 2025-02-15 03:14:34,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:14:34,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:14:34,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:14:34,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37961.92 MB 2025-02-15 03:14:34,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38728.92 MB 2025-02-15 03:14:34,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:14:34,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44495.27 MB 2025-02-15 03:14:34,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44904.22 MB 2025-02-15 03:14:34,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 03:14:34,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39436.71 MB 2025-02-15 03:14:34,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:14:34,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:14:34,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:14:34,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39141.81 MB 2025-02-15 03:14:34,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39370.19 MB 2025-02-15 03:14:34,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.38 MB 2025-02-15 03:14:34,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44904.22 MB 2025-02-15 03:14:34,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44904.22 MB 2025-02-15 03:14:34,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:14:34,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39586.78 MB 2025-02-15 03:14:34,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:14:34,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:14:34,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.94 seconds 2025-02-15 03:14:34,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25395.76 MB 2025-02-15 03:14:34,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39571.04 MB 2025-02-15 03:14:34,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14175.28 MB 2025-02-15 03:14:34,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59764.64 MB 2025-02-15 03:14:34,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44904.22 MB 2025-02-15 03:14:34,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14860.42 MB 2025-02-15 03:14:34,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39586.78 MB 2025-02-15 03:14:34,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:14:34,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:14:34,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:14:34,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39571.04 MB 2025-02-15 03:14:34,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30384.16 MB 2025-02-15 03:14:34,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9186.88 MB 2025-02-15 03:14:34,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44904.22 MB 2025-02-15 03:14:34,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44904.22 MB 2025-02-15 03:14:34,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:14:34,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42068.88 MB 2025-02-15 03:14:34,708 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 03:14:34,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:14:34,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:14:34,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:14:34,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:14:34,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:14:34,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30384.16 MB 2025-02-15 03:14:34,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38776.40 MB 2025-02-15 03:14:34,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.24 MB 2025-02-15 03:14:34,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44904.22 MB 2025-02-15 03:14:34,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49077.55 MB 2025-02-15 03:14:34,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 03:14:34,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38776.40 MB 2025-02-15 03:14:34,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 03:14:34,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:14:34,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:14:34,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:14:34,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:14:34,892 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:14:34,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:14:34,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:14:34,893 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:15:03,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:03,400 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:15:03,407 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:15:03,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:03,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:15:03,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:03,415 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:15:06,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:15:06,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:15:06,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.54 seconds 2025-02-15 03:15:06,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:06,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19713.22 MB 2025-02-15 03:15:06,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20502.41 MB 2025-02-15 03:15:06,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-15 03:15:06,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57420.02 MB 2025-02-15 03:15:06,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:06,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27013.41 MB 2025-02-15 03:15:06,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29411.09 MB 2025-02-15 03:15:06,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:15:06,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:15:06,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:15:06,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:06,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20502.41 MB 2025-02-15 03:15:06,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20758.29 MB 2025-02-15 03:15:06,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.88 MB 2025-02-15 03:15:06,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:06,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:06,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:06,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23381.85 MB 2025-02-15 03:15:07,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:15:07,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:15:07,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-15 03:15:07,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:07,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20758.29 MB 2025-02-15 03:15:07,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21030.34 MB 2025-02-15 03:15:07,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.06 MB 2025-02-15 03:15:07,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:07,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:07,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:07,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25012.87 MB 2025-02-15 03:15:08,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:15:08,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:15:08,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:15:08,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21030.34 MB 2025-02-15 03:15:08,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21998.49 MB 2025-02-15 03:15:08,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 968.15 MB 2025-02-15 03:15:08,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:08,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:08,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:08,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22724.93 MB 2025-02-15 03:15:08,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:15:08,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:15:08,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 03:15:08,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21998.49 MB 2025-02-15 03:15:08,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23147.48 MB 2025-02-15 03:15:08,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1148.98 MB 2025-02-15 03:15:08,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:08,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:08,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:08,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.89 MB 2025-02-15 03:15:08,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:15:08,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:15:08,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:15:08,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21030.34 MB 2025-02-15 03:15:08,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23147.48 MB 2025-02-15 03:15:08,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2117.13 MB 2025-02-15 03:15:08,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:08,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 03:15:08,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:08,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.89 MB 2025-02-15 03:15:08,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:15:08,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:15:08,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 03:15:08,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23933.42 MB 2025-02-15 03:15:08,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24326.50 MB 2025-02-15 03:15:08,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 393.09 MB 2025-02-15 03:15:08,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 03:15:08,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30614.22 MB 2025-02-15 03:15:08,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-15 03:15:08,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24689.31 MB 2025-02-15 03:15:08,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:15:08,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:15:08,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:15:08,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24538.12 MB 2025-02-15 03:15:08,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24755.05 MB 2025-02-15 03:15:08,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.94 MB 2025-02-15 03:15:08,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30614.22 MB 2025-02-15 03:15:08,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30614.22 MB 2025-02-15 03:15:08,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:08,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24789.99 MB 2025-02-15 03:15:08,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:15:08,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:15:08,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.89 seconds 2025-02-15 03:15:08,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18936.27 MB 2025-02-15 03:15:08,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24955.88 MB 2025-02-15 03:15:08,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6019.61 MB 2025-02-15 03:15:08,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57420.02 MB 2025-02-15 03:15:08,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30614.22 MB 2025-02-15 03:15:08,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26805.80 MB 2025-02-15 03:15:08,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24955.88 MB 2025-02-15 03:15:08,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:15:08,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:15:08,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 03:15:08,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20006.24 MB 2025-02-15 03:15:08,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23017.03 MB 2025-02-15 03:15:08,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.79 MB 2025-02-15 03:15:08,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30614.22 MB 2025-02-15 03:15:08,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30614.22 MB 2025-02-15 03:15:08,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:15:08,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23318.03 MB 2025-02-15 03:15:08,612 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 03:15:08,612 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:15:08,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:15:08,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:15:08,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:15:08,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:15:08,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23017.03 MB 2025-02-15 03:15:08,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31445.40 MB 2025-02-15 03:15:08,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8428.37 MB 2025-02-15 03:15:08,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30614.22 MB 2025-02-15 03:15:08,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34804.33 MB 2025-02-15 03:15:08,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 03:15:08,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31445.40 MB 2025-02-15 03:15:08,870 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 03:15:08,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:08,872 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:15:08,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:08,874 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:15:08,882 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:15:08,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:15:08,884 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:15:08,884 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:16:30,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:30,359 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:16:30,365 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:16:30,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:30,370 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 566, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:16:30,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:30,371 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 566, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:16:39,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:16:39,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:16:39,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.68 seconds 2025-02-15 03:16:39,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:39,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22103.30 MB 2025-02-15 03:16:39,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24106.34 MB 2025-02-15 03:16:39,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2003.04 MB 2025-02-15 03:16:39,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43184.55 MB 2025-02-15 03:16:39,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26864.52 MB 2025-02-15 03:16:39,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16320.04 MB 2025-02-15 03:16:39,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32934.43 MB 2025-02-15 03:16:39,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:16:39,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:16:39,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:16:39,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:39,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24106.34 MB 2025-02-15 03:16:39,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23911.97 MB 2025-02-15 03:16:39,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -194.37 MB 2025-02-15 03:16:39,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26864.52 MB 2025-02-15 03:16:39,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32967.23 MB 2025-02-15 03:16:39,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6102.71 MB 2025-02-15 03:16:39,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32308.64 MB 2025-02-15 03:16:41,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:16:41,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:16:41,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:16:41,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23911.97 MB 2025-02-15 03:16:41,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24442.81 MB 2025-02-15 03:16:41,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:16:41,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32967.23 MB 2025-02-15 03:16:41,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26984.05 MB 2025-02-15 03:16:41,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5983.17 MB 2025-02-15 03:16:41,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28422.40 MB 2025-02-15 03:16:41,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:16:41,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:16:41,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:16:41,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24442.81 MB 2025-02-15 03:16:41,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26332.35 MB 2025-02-15 03:16:41,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:16:41,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26984.05 MB 2025-02-15 03:16:41,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29815.21 MB 2025-02-15 03:16:41,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:16:41,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27749.78 MB 2025-02-15 03:16:41,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:16:41,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:16:41,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:16:41,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26332.35 MB 2025-02-15 03:16:41,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28575.25 MB 2025-02-15 03:16:41,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 03:16:41,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29815.21 MB 2025-02-15 03:16:41,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 03:16:41,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 03:16:41,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34119.54 MB 2025-02-15 03:16:41,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:16:41,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:16:41,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:16:41,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24442.81 MB 2025-02-15 03:16:41,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28575.25 MB 2025-02-15 03:16:41,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 03:16:41,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26984.05 MB 2025-02-15 03:16:41,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 03:16:41,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-15 03:16:41,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34119.54 MB 2025-02-15 03:16:41,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:16:41,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:16:41,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:16:41,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30108.80 MB 2025-02-15 03:16:41,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30875.80 MB 2025-02-15 03:16:41,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:16:41,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36186.36 MB 2025-02-15 03:16:41,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-15 03:16:41,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 03:16:41,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31583.59 MB 2025-02-15 03:16:41,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:16:41,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:16:41,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:16:41,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31288.69 MB 2025-02-15 03:16:41,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31517.91 MB 2025-02-15 03:16:41,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.22 MB 2025-02-15 03:16:41,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36597.40 MB 2025-02-15 03:16:41,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-15 03:16:41,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:16:41,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31702.26 MB 2025-02-15 03:16:41,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:16:41,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:16:41,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.09 seconds 2025-02-15 03:16:41,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20131.31 MB 2025-02-15 03:16:41,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31718.98 MB 2025-02-15 03:16:41,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11587.67 MB 2025-02-15 03:16:41,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43184.55 MB 2025-02-15 03:16:41,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-15 03:16:41,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6587.15 MB 2025-02-15 03:16:41,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31718.98 MB 2025-02-15 03:16:41,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:16:41,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:16:41,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:16:41,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31718.98 MB 2025-02-15 03:16:41,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25135.70 MB 2025-02-15 03:16:41,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6583.28 MB 2025-02-15 03:16:41,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36597.40 MB 2025-02-15 03:16:41,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-15 03:16:41,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:16:41,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34230.65 MB 2025-02-15 03:16:41,746 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:16:41,746 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:16:41,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:16:41,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:16:41,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:16:41,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:16:41,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25135.70 MB 2025-02-15 03:16:41,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33574.72 MB 2025-02-15 03:16:41,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:16:41,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36597.40 MB 2025-02-15 03:16:41,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44988.10 MB 2025-02-15 03:16:41,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:16:41,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33574.72 MB 2025-02-15 03:16:41,917 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:16:41,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:41,918 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:16:41,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:41,919 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:16:41,924 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:16:41,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:16:41,925 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:16:41,925 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:18:02,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:02,535 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:18:02,544 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:18:02,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:02,554 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1869, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:18:02,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:02,556 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1869, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:18:31,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:18:31,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:18:31,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.82 seconds 2025-02-15 03:18:31,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:31,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31182.81 MB 2025-02-15 03:18:31,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37797.23 MB 2025-02-15 03:18:31,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6614.42 MB 2025-02-15 03:18:31,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57573.11 MB 2025-02-15 03:18:31,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44054.87 MB 2025-02-15 03:18:31,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13518.24 MB 2025-02-15 03:18:31,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46770.29 MB 2025-02-15 03:18:31,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:18:31,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:18:31,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:18:31,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:31,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37797.23 MB 2025-02-15 03:18:31,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30684.81 MB 2025-02-15 03:18:31,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7112.42 MB 2025-02-15 03:18:31,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44054.87 MB 2025-02-15 03:18:31,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57877.20 MB 2025-02-15 03:18:31,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13822.33 MB 2025-02-15 03:18:31,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56452.29 MB 2025-02-15 03:18:33,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:18:33,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:18:33,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:18:33,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30684.81 MB 2025-02-15 03:18:33,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31215.66 MB 2025-02-15 03:18:33,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:18:33,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57877.20 MB 2025-02-15 03:18:33,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-15 03:18:33,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23215.47 MB 2025-02-15 03:18:33,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35194.20 MB 2025-02-15 03:18:33,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:18:33,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:18:33,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:18:33,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31215.66 MB 2025-02-15 03:18:33,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33105.19 MB 2025-02-15 03:18:33,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:18:33,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-15 03:18:33,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37492.88 MB 2025-02-15 03:18:33,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:18:33,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34522.62 MB 2025-02-15 03:18:33,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:18:33,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:18:33,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:18:33,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33105.19 MB 2025-02-15 03:18:33,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35347.05 MB 2025-02-15 03:18:33,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:18:33,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37492.88 MB 2025-02-15 03:18:33,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43627.05 MB 2025-02-15 03:18:33,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:18:33,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40891.33 MB 2025-02-15 03:18:33,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:18:33,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:18:33,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:18:33,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31215.66 MB 2025-02-15 03:18:33,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35347.05 MB 2025-02-15 03:18:33,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:18:33,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-15 03:18:33,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43627.05 MB 2025-02-15 03:18:33,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 03:18:33,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40891.33 MB 2025-02-15 03:18:33,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:18:33,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:18:33,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:18:33,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36880.59 MB 2025-02-15 03:18:33,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37647.59 MB 2025-02-15 03:18:33,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:18:33,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43627.05 MB 2025-02-15 03:18:33,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44040.19 MB 2025-02-15 03:18:33,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:18:33,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38355.38 MB 2025-02-15 03:18:33,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:18:33,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:18:33,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:18:33,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38060.48 MB 2025-02-15 03:18:33,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38288.59 MB 2025-02-15 03:18:33,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.11 MB 2025-02-15 03:18:33,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44040.19 MB 2025-02-15 03:18:33,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44040.19 MB 2025-02-15 03:18:33,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:18:33,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38490.85 MB 2025-02-15 03:18:33,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:18:33,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:18:33,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.32 seconds 2025-02-15 03:18:33,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:33,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24671.07 MB 2025-02-15 03:18:33,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38489.44 MB 2025-02-15 03:18:33,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13818.38 MB 2025-02-15 03:18:33,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57573.11 MB 2025-02-15 03:18:33,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44040.19 MB 2025-02-15 03:18:33,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13532.92 MB 2025-02-15 03:18:33,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38490.85 MB 2025-02-15 03:18:34,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:18:34,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:18:34,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:18:34,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:34,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38489.44 MB 2025-02-15 03:18:34,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29662.22 MB 2025-02-15 03:18:34,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8827.22 MB 2025-02-15 03:18:34,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44040.19 MB 2025-02-15 03:18:34,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44040.19 MB 2025-02-15 03:18:34,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:18:34,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40989.74 MB 2025-02-15 03:18:34,168 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 03:18:34,168 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:18:34,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:18:34,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:18:34,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:18:34,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:18:34,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29662.22 MB 2025-02-15 03:18:34,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38063.16 MB 2025-02-15 03:18:34,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-15 03:18:34,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44040.19 MB 2025-02-15 03:18:34,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52393.15 MB 2025-02-15 03:18:34,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-15 03:18:34,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38063.16 MB 2025-02-15 03:18:34,335 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 03:18:34,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:34,337 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:18:34,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:34,337 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:18:34,342 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:18:34,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:18:34,343 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:18:34,343 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:20:34,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:20:34,810 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:20:34,815 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:20:34,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:20:34,819 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:20:34,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:20:34,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:21:02,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:21:02,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:21:02,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.67 seconds 2025-02-15 03:21:02,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:02,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30708.98 MB 2025-02-15 03:21:02,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37082.62 MB 2025-02-15 03:21:02,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-15 03:21:02,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64921.53 MB 2025-02-15 03:21:02,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43750.79 MB 2025-02-15 03:21:02,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21170.75 MB 2025-02-15 03:21:02,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46069.96 MB 2025-02-15 03:21:02,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:21:02,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:21:02,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 03:21:02,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:02,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37082.62 MB 2025-02-15 03:21:02,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30331.30 MB 2025-02-15 03:21:02,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-15 03:21:02,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43750.79 MB 2025-02-15 03:21:02,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57185.14 MB 2025-02-15 03:21:02,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13434.36 MB 2025-02-15 03:21:02,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55157.52 MB 2025-02-15 03:21:04,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:21:04,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:21:04,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:21:04,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30331.30 MB 2025-02-15 03:21:04,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30862.15 MB 2025-02-15 03:21:04,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:21:04,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57185.14 MB 2025-02-15 03:21:04,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34615.59 MB 2025-02-15 03:21:04,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22569.55 MB 2025-02-15 03:21:04,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34841.73 MB 2025-02-15 03:21:04,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:21:04,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:21:04,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:21:04,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30862.15 MB 2025-02-15 03:21:04,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32751.68 MB 2025-02-15 03:21:04,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:21:04,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34615.59 MB 2025-02-15 03:21:04,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 03:21:04,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 03:21:04,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34169.11 MB 2025-02-15 03:21:04,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:21:04,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:21:04,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:21:04,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32751.68 MB 2025-02-15 03:21:04,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34993.54 MB 2025-02-15 03:21:04,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:21:04,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 03:21:04,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-15 03:21:04,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:21:04,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40537.82 MB 2025-02-15 03:21:04,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:21:04,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:21:04,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:21:04,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30862.15 MB 2025-02-15 03:21:04,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34993.54 MB 2025-02-15 03:21:04,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:21:04,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34615.59 MB 2025-02-15 03:21:04,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-15 03:21:04,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 03:21:04,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40537.82 MB 2025-02-15 03:21:04,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:21:04,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:21:04,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:21:04,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36527.08 MB 2025-02-15 03:21:04,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37294.08 MB 2025-02-15 03:21:04,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:21:04,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43109.06 MB 2025-02-15 03:21:04,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43518.00 MB 2025-02-15 03:21:04,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 03:21:04,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38001.87 MB 2025-02-15 03:21:04,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:21:04,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:21:04,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:21:04,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37706.97 MB 2025-02-15 03:21:04,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37935.88 MB 2025-02-15 03:21:04,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 03:21:04,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43518.00 MB 2025-02-15 03:21:04,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43518.00 MB 2025-02-15 03:21:04,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:21:04,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38129.00 MB 2025-02-15 03:21:04,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:21:04,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:21:04,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.14 seconds 2025-02-15 03:21:04,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:04,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24434.15 MB 2025-02-15 03:21:04,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38136.71 MB 2025-02-15 03:21:04,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13702.56 MB 2025-02-15 03:21:04,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64921.53 MB 2025-02-15 03:21:04,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43518.00 MB 2025-02-15 03:21:04,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21403.53 MB 2025-02-15 03:21:04,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38136.71 MB 2025-02-15 03:21:05,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:21:05,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:21:05,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:21:05,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:05,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38136.71 MB 2025-02-15 03:21:05,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29434.76 MB 2025-02-15 03:21:05,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8701.95 MB 2025-02-15 03:21:05,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43518.00 MB 2025-02-15 03:21:05,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43518.00 MB 2025-02-15 03:21:05,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:21:05,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40645.33 MB 2025-02-15 03:21:05,253 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 03:21:05,253 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:21:05,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:21:05,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:21:05,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:21:05,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:05,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29434.76 MB 2025-02-15 03:21:05,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37863.88 MB 2025-02-15 03:21:05,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 03:21:05,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43518.00 MB 2025-02-15 03:21:05,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51898.22 MB 2025-02-15 03:21:05,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 03:21:05,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37863.88 MB 2025-02-15 03:21:05,420 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 03:21:05,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:05,422 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:21:05,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:05,423 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:21:05,428 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:21:05,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:05,429 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:21:05,429 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:21:17,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:17,374 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:21:17,379 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:21:17,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:17,383 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2583, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:21:17,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:21:17,384 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2583, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:21:57,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:21:57,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:21:57,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.18 seconds 2025-02-15 03:21:57,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:57,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36158.79 MB 2025-02-15 03:21:57,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45300.27 MB 2025-02-15 03:21:57,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9141.49 MB 2025-02-15 03:21:57,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78280.39 MB 2025-02-15 03:21:57,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47815.07 MB 2025-02-15 03:21:57,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30465.33 MB 2025-02-15 03:21:57,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54441.36 MB 2025-02-15 03:21:57,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:21:57,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:21:57,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 03:21:57,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:57,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45300.27 MB 2025-02-15 03:21:57,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34398.43 MB 2025-02-15 03:21:57,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10901.84 MB 2025-02-15 03:21:57,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47815.07 MB 2025-02-15 03:21:57,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67450.70 MB 2025-02-15 03:21:57,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19635.63 MB 2025-02-15 03:21:57,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 71635.06 MB 2025-02-15 03:21:59,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:21:59,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:21:59,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:21:59,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:59,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34398.43 MB 2025-02-15 03:21:59,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34929.27 MB 2025-02-15 03:21:59,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:21:59,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67450.70 MB 2025-02-15 03:21:59,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36758.88 MB 2025-02-15 03:21:59,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30691.82 MB 2025-02-15 03:21:59,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38907.82 MB 2025-02-15 03:21:59,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:21:59,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:21:59,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:21:59,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:59,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34929.27 MB 2025-02-15 03:21:59,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36818.55 MB 2025-02-15 03:21:59,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 03:21:59,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36758.88 MB 2025-02-15 03:21:59,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40533.75 MB 2025-02-15 03:21:59,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:21:59,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38235.97 MB 2025-02-15 03:21:59,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:21:59,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:21:59,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:21:59,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:59,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36818.55 MB 2025-02-15 03:21:59,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39060.40 MB 2025-02-15 03:21:59,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:21:59,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40533.75 MB 2025-02-15 03:21:59,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46667.92 MB 2025-02-15 03:21:59,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:21:59,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44604.68 MB 2025-02-15 03:21:59,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:21:59,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:21:59,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:21:59,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:21:59,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34929.27 MB 2025-02-15 03:21:59,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39060.40 MB 2025-02-15 03:21:59,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 03:21:59,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36758.88 MB 2025-02-15 03:21:59,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46667.92 MB 2025-02-15 03:21:59,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 03:21:59,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44604.68 MB 2025-02-15 03:22:00,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:22:00,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:22:00,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:22:00,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:00,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40593.94 MB 2025-02-15 03:22:00,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41360.95 MB 2025-02-15 03:22:00,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:22:00,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46667.92 MB 2025-02-15 03:22:00,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47076.87 MB 2025-02-15 03:22:00,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 03:22:00,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42068.73 MB 2025-02-15 03:22:00,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:22:00,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:22:00,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:22:00,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:00,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41773.83 MB 2025-02-15 03:22:00,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42001.63 MB 2025-02-15 03:22:00,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.79 MB 2025-02-15 03:22:00,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47076.87 MB 2025-02-15 03:22:00,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47076.87 MB 2025-02-15 03:22:00,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:22:00,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42224.67 MB 2025-02-15 03:22:00,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:22:00,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:22:00,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.78 seconds 2025-02-15 03:22:00,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:00,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27159.05 MB 2025-02-15 03:22:00,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42202.48 MB 2025-02-15 03:22:00,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15043.43 MB 2025-02-15 03:22:00,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69279.42 MB 2025-02-15 03:22:00,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47076.87 MB 2025-02-15 03:22:00,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22202.55 MB 2025-02-15 03:22:00,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42224.67 MB 2025-02-15 03:22:00,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:22:00,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:22:00,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:22:00,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:00,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42202.48 MB 2025-02-15 03:22:00,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32158.95 MB 2025-02-15 03:22:00,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10043.53 MB 2025-02-15 03:22:00,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47076.87 MB 2025-02-15 03:22:00,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47076.87 MB 2025-02-15 03:22:00,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:22:00,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44710.46 MB 2025-02-15 03:22:00,455 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 03:22:00,455 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:22:00,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:22:00,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:22:00,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:22:00,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:00,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32158.95 MB 2025-02-15 03:22:00,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40585.25 MB 2025-02-15 03:22:00,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.30 MB 2025-02-15 03:22:00,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47076.87 MB 2025-02-15 03:22:00,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51266.98 MB 2025-02-15 03:22:00,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 03:22:00,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40585.25 MB 2025-02-15 03:22:00,626 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 03:22:00,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:00,628 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:22:00,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:00,629 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:22:00,633 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:22:00,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:00,634 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:22:00,634 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:22:13,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:13,680 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:22:13,685 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:22:13,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:13,688 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 213, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:22:13,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:13,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 213, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:22:17,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:22:17,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:22:17,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.35 seconds 2025-02-15 03:22:17,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:17,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19643.54 MB 2025-02-15 03:22:17,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20397.34 MB 2025-02-15 03:22:17,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 753.80 MB 2025-02-15 03:22:17,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63833.11 MB 2025-02-15 03:22:17,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23372.76 MB 2025-02-15 03:22:17,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40460.35 MB 2025-02-15 03:22:17,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29342.21 MB 2025-02-15 03:22:17,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:22:17,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:22:17,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:22:17,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:17,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20397.34 MB 2025-02-15 03:22:17,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20742.06 MB 2025-02-15 03:22:17,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.72 MB 2025-02-15 03:22:17,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23372.76 MB 2025-02-15 03:22:17,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25595.74 MB 2025-02-15 03:22:17,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2222.98 MB 2025-02-15 03:22:17,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23347.65 MB 2025-02-15 03:22:18,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:22:18,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:22:18,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-15 03:22:18,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20742.06 MB 2025-02-15 03:22:18,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21020.75 MB 2025-02-15 03:22:18,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.69 MB 2025-02-15 03:22:18,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25595.74 MB 2025-02-15 03:22:18,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23624.42 MB 2025-02-15 03:22:18,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1971.32 MB 2025-02-15 03:22:18,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24997.68 MB 2025-02-15 03:22:18,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:22:18,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:22:18,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:22:18,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21020.75 MB 2025-02-15 03:22:18,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22012.51 MB 2025-02-15 03:22:18,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 991.76 MB 2025-02-15 03:22:18,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23624.42 MB 2025-02-15 03:22:18,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24618.47 MB 2025-02-15 03:22:18,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 994.05 MB 2025-02-15 03:22:18,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22756.67 MB 2025-02-15 03:22:18,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:22:18,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:22:18,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:22:18,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22012.51 MB 2025-02-15 03:22:18,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23189.52 MB 2025-02-15 03:22:18,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1177.01 MB 2025-02-15 03:22:18,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24618.47 MB 2025-02-15 03:22:18,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27602.71 MB 2025-02-15 03:22:18,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2984.25 MB 2025-02-15 03:22:18,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26100.23 MB 2025-02-15 03:22:18,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:22:18,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:22:18,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 03:22:18,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21020.75 MB 2025-02-15 03:22:18,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23189.52 MB 2025-02-15 03:22:18,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2168.77 MB 2025-02-15 03:22:18,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23624.42 MB 2025-02-15 03:22:18,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27602.71 MB 2025-02-15 03:22:18,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3978.30 MB 2025-02-15 03:22:18,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26100.23 MB 2025-02-15 03:22:18,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:22:18,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:22:18,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:22:18,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23994.63 MB 2025-02-15 03:22:18,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24397.30 MB 2025-02-15 03:22:18,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.68 MB 2025-02-15 03:22:18,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27602.71 MB 2025-02-15 03:22:18,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27814.53 MB 2025-02-15 03:22:18,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 211.81 MB 2025-02-15 03:22:18,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24770.51 MB 2025-02-15 03:22:18,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:22:18,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:22:18,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:22:18,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24614.08 MB 2025-02-15 03:22:18,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24818.89 MB 2025-02-15 03:22:18,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.81 MB 2025-02-15 03:22:18,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27814.53 MB 2025-02-15 03:22:18,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27818.72 MB 2025-02-15 03:22:18,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 03:22:18,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24877.88 MB 2025-02-15 03:22:18,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:22:18,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:22:18,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.60 seconds 2025-02-15 03:22:18,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18901.43 MB 2025-02-15 03:22:18,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25019.96 MB 2025-02-15 03:22:18,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6118.53 MB 2025-02-15 03:22:18,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63833.11 MB 2025-02-15 03:22:18,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27818.72 MB 2025-02-15 03:22:18,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36014.39 MB 2025-02-15 03:22:18,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25019.96 MB 2025-02-15 03:22:18,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:22:18,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:22:18,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:22:18,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19994.60 MB 2025-02-15 03:22:18,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23008.63 MB 2025-02-15 03:22:18,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 03:22:18,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27818.72 MB 2025-02-15 03:22:18,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27818.72 MB 2025-02-15 03:22:18,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:22:18,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23310.00 MB 2025-02-15 03:22:18,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:22:18,591 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:22:18,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:22:18,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:22:18,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 03:22:18,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:22:18,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23008.63 MB 2025-02-15 03:22:18,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31447.65 MB 2025-02-15 03:22:18,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:22:18,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27818.72 MB 2025-02-15 03:22:18,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38308.68 MB 2025-02-15 03:22:18,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 03:22:18,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31447.65 MB 2025-02-15 03:22:18,778 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:22:18,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:18,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:22:18,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:18,780 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:22:18,785 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:22:18,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:22:18,786 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:22:18,786 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:23:49,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:49,672 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:23:49,681 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:23:49,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:49,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 224, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:23:49,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:49,691 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 224, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:23:53,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:23:53,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:23:53,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-15 03:23:53,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:53,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19720.19 MB 2025-02-15 03:23:53,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20512.91 MB 2025-02-15 03:23:53,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 792.72 MB 2025-02-15 03:23:53,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50893.68 MB 2025-02-15 03:23:53,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23397.92 MB 2025-02-15 03:23:53,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27495.76 MB 2025-02-15 03:23:53,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29418.86 MB 2025-02-15 03:23:53,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:23:53,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:23:53,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:23:53,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:53,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20512.91 MB 2025-02-15 03:23:53,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20568.16 MB 2025-02-15 03:23:53,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 55.25 MB 2025-02-15 03:23:53,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23397.92 MB 2025-02-15 03:23:53,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24956.11 MB 2025-02-15 03:23:53,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1558.18 MB 2025-02-15 03:23:53,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23001.04 MB 2025-02-15 03:23:54,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:23:54,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:23:54,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 03:23:54,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20568.16 MB 2025-02-15 03:23:54,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20803.06 MB 2025-02-15 03:23:54,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 03:23:54,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24956.11 MB 2025-02-15 03:23:54,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-15 03:23:54,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1891.63 MB 2025-02-15 03:23:54,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24739.96 MB 2025-02-15 03:23:54,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:23:54,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:23:54,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:23:54,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20802.99 MB 2025-02-15 03:23:54,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21638.91 MB 2025-02-15 03:23:54,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 03:23:54,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-15 03:23:54,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23903.34 MB 2025-02-15 03:23:54,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 838.86 MB 2025-02-15 03:23:54,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22266.13 MB 2025-02-15 03:23:54,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:23:54,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:23:54,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:23:54,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21638.91 MB 2025-02-15 03:23:54,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22630.97 MB 2025-02-15 03:23:54,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 03:23:54,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23903.34 MB 2025-02-15 03:23:54,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26839.35 MB 2025-02-15 03:23:54,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-15 03:23:54,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25086.11 MB 2025-02-15 03:23:54,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:23:54,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:23:54,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:23:54,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20802.99 MB 2025-02-15 03:23:54,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22630.97 MB 2025-02-15 03:23:54,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 03:23:54,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-15 03:23:54,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26839.35 MB 2025-02-15 03:23:54,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:23:54,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25086.11 MB 2025-02-15 03:23:54,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:23:54,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:23:54,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:23:54,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23309.56 MB 2025-02-15 03:23:54,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23650.79 MB 2025-02-15 03:23:54,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.23 MB 2025-02-15 03:23:54,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26839.35 MB 2025-02-15 03:23:54,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 03:23:54,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 03:23:54,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23969.23 MB 2025-02-15 03:23:54,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:23:54,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:23:54,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:23:54,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23833.50 MB 2025-02-15 03:23:54,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24062.35 MB 2025-02-15 03:23:54,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-15 03:23:54,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 03:23:54,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 03:23:54,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:23:54,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24082.09 MB 2025-02-15 03:23:54,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:23:54,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:23:54,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.55 seconds 2025-02-15 03:23:54,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18939.76 MB 2025-02-15 03:23:54,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24263.39 MB 2025-02-15 03:23:54,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5323.64 MB 2025-02-15 03:23:54,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50893.68 MB 2025-02-15 03:23:54,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 03:23:54,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23873.98 MB 2025-02-15 03:23:54,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24263.39 MB 2025-02-15 03:23:54,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:23:54,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:23:54,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:23:54,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24263.39 MB 2025-02-15 03:23:54,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22892.94 MB 2025-02-15 03:23:54,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1370.46 MB 2025-02-15 03:23:54,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 03:23:54,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 03:23:54,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:23:54,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24497.79 MB 2025-02-15 03:23:54,526 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 03:23:54,526 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:23:54,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:23:54,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:23:54,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:23:54,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:23:54,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22892.94 MB 2025-02-15 03:23:54,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31331.77 MB 2025-02-15 03:23:54,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 03:23:54,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 03:23:54,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37505.47 MB 2025-02-15 03:23:54,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 03:23:54,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31331.77 MB 2025-02-15 03:23:54,693 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 03:23:54,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:54,695 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:23:54,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:54,696 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:23:54,700 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:23:54,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:23:54,701 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:23:54,701 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:24:00,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:00,994 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:24:01,002 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:24:01,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:01,008 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:24:01,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:01,010 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:24:35,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:24:35,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:24:35,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.23 seconds 2025-02-15 03:24:35,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:35,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33586.83 MB 2025-02-15 03:24:35,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41422.05 MB 2025-02-15 03:24:35,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7835.22 MB 2025-02-15 03:24:35,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45894.07 MB 2025-02-15 03:24:35,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45283.80 MB 2025-02-15 03:24:35,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -610.27 MB 2025-02-15 03:24:35,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50306.76 MB 2025-02-15 03:24:35,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:24:35,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:24:35,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:24:35,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:35,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41422.05 MB 2025-02-15 03:24:35,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32479.41 MB 2025-02-15 03:24:35,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8942.64 MB 2025-02-15 03:24:35,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45283.80 MB 2025-02-15 03:24:35,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 03:24:35,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17465.08 MB 2025-02-15 03:24:35,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64493.74 MB 2025-02-15 03:24:37,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:24:37,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:24:37,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:24:37,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32479.41 MB 2025-02-15 03:24:37,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33010.25 MB 2025-02-15 03:24:37,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:24:37,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 03:24:37,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35909.53 MB 2025-02-15 03:24:37,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26839.35 MB 2025-02-15 03:24:37,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36988.80 MB 2025-02-15 03:24:37,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:24:37,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:24:37,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:24:37,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33010.25 MB 2025-02-15 03:24:37,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34899.53 MB 2025-02-15 03:24:37,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 03:24:37,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35909.53 MB 2025-02-15 03:24:37,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39684.41 MB 2025-02-15 03:24:37,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:24:37,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36316.95 MB 2025-02-15 03:24:37,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:24:37,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:24:37,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:24:37,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34899.53 MB 2025-02-15 03:24:37,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37141.38 MB 2025-02-15 03:24:37,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:24:37,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39684.41 MB 2025-02-15 03:24:37,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45818.58 MB 2025-02-15 03:24:37,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:24:37,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42685.66 MB 2025-02-15 03:24:37,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:24:37,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:24:37,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:24:37,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33010.25 MB 2025-02-15 03:24:37,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37141.38 MB 2025-02-15 03:24:37,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 03:24:37,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35909.53 MB 2025-02-15 03:24:37,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45818.58 MB 2025-02-15 03:24:37,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 03:24:37,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42685.66 MB 2025-02-15 03:24:37,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:24:37,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:24:37,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:24:37,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38674.92 MB 2025-02-15 03:24:37,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39441.93 MB 2025-02-15 03:24:37,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:24:37,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45818.58 MB 2025-02-15 03:24:37,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-15 03:24:37,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 03:24:37,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40149.71 MB 2025-02-15 03:24:37,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:24:37,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:24:37,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:24:37,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39854.81 MB 2025-02-15 03:24:37,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40083.80 MB 2025-02-15 03:24:37,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-15 03:24:37,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46229.62 MB 2025-02-15 03:24:37,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-15 03:24:37,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:37,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40304.10 MB 2025-02-15 03:24:37,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:24:37,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:24:37,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.78 seconds 2025-02-15 03:24:37,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:37,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25873.08 MB 2025-02-15 03:24:37,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40284.70 MB 2025-02-15 03:24:37,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14411.62 MB 2025-02-15 03:24:37,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45894.07 MB 2025-02-15 03:24:37,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-15 03:24:37,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 335.54 MB 2025-02-15 03:24:37,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40304.10 MB 2025-02-15 03:24:38,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:24:38,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:24:38,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:24:38,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:38,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40284.70 MB 2025-02-15 03:24:38,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30874.80 MB 2025-02-15 03:24:38,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9409.90 MB 2025-02-15 03:24:38,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46229.62 MB 2025-02-15 03:24:38,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-15 03:24:38,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:38,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42794.22 MB 2025-02-15 03:24:38,085 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 03:24:38,085 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 03:24:38,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:24:38,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:24:38,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:24:38,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:38,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30874.80 MB 2025-02-15 03:24:38,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39306.26 MB 2025-02-15 03:24:38,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 03:24:38,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46229.62 MB 2025-02-15 03:24:38,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54614.03 MB 2025-02-15 03:24:38,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 03:24:38,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39306.26 MB 2025-02-15 03:24:38,256 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 03:24:38,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:38,258 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:24:38,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:38,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:24:38,263 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:24:38,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:38,265 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:24:38,265 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 03:24:47,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:47,519 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:24:47,523 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:24:47,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:47,527 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 93, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:24:47,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:47,528 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 93, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:24:49,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:24:49,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:24:49,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.48 seconds 2025-02-15 03:24:49,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18807.36 MB 2025-02-15 03:24:49,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19136.48 MB 2025-02-15 03:24:49,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 329.12 MB 2025-02-15 03:24:49,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62998.45 MB 2025-02-15 03:24:49,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38134.61 MB 2025-02-15 03:24:49,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28052.24 MB 2025-02-15 03:24:49,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:24:49,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:24:49,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:24:49,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19136.48 MB 2025-02-15 03:24:49,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19295.94 MB 2025-02-15 03:24:49,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.46 MB 2025-02-15 03:24:49,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19789.69 MB 2025-02-15 03:24:49,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:24:49,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:24:49,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.45 seconds 2025-02-15 03:24:49,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19295.94 MB 2025-02-15 03:24:49,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19419.36 MB 2025-02-15 03:24:49,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.42 MB 2025-02-15 03:24:49,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23380.66 MB 2025-02-15 03:24:49,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:24:49,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:24:49,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:24:49,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19419.30 MB 2025-02-15 03:24:49,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19858.51 MB 2025-02-15 03:24:49,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.21 MB 2025-02-15 03:24:49,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20188.07 MB 2025-02-15 03:24:49,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:24:49,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:24:49,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 03:24:49,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19858.51 MB 2025-02-15 03:24:49,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20391.98 MB 2025-02-15 03:24:49,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 533.47 MB 2025-02-15 03:24:49,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21668.79 MB 2025-02-15 03:24:49,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:24:49,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:24:49,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:24:49,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19419.30 MB 2025-02-15 03:24:49,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20391.98 MB 2025-02-15 03:24:49,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 972.68 MB 2025-02-15 03:24:49,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 03:24:49,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21668.79 MB 2025-02-15 03:24:49,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:24:49,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:24:49,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 03:24:49,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20907.78 MB 2025-02-15 03:24:49,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21131.82 MB 2025-02-15 03:24:49,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.04 MB 2025-02-15 03:24:49,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 03:24:49,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25000.15 MB 2025-02-15 03:24:49,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-15 03:24:49,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21296.38 MB 2025-02-15 03:24:49,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:24:49,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:24:49,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:24:49,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21273.83 MB 2025-02-15 03:24:49,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21495.87 MB 2025-02-15 03:24:49,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.03 MB 2025-02-15 03:24:49,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25000.15 MB 2025-02-15 03:24:49,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25000.15 MB 2025-02-15 03:24:49,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21495.87 MB 2025-02-15 03:24:49,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:24:49,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:24:49,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.08 seconds 2025-02-15 03:24:49,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18483.34 MB 2025-02-15 03:24:49,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21693.82 MB 2025-02-15 03:24:49,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3210.47 MB 2025-02-15 03:24:49,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62998.45 MB 2025-02-15 03:24:49,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25000.15 MB 2025-02-15 03:24:49,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37998.30 MB 2025-02-15 03:24:49,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21693.82 MB 2025-02-15 03:24:49,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:24:49,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:24:49,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:24:49,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19023.30 MB 2025-02-15 03:24:49,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21991.44 MB 2025-02-15 03:24:49,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2968.14 MB 2025-02-15 03:24:49,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25000.15 MB 2025-02-15 03:24:49,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25000.15 MB 2025-02-15 03:24:49,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:24:49,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22288.12 MB 2025-02-15 03:24:49,897 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8035, cut from 8037 2025-02-15 03:24:49,897 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2 ('] 2025-02-15 03:24:49,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:24:49,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:24:49,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:24:49,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:24:49,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21991.44 MB 2025-02-15 03:24:49,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30299.51 MB 2025-02-15 03:24:49,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8308.08 MB 2025-02-15 03:24:49,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25000.15 MB 2025-02-15 03:24:49,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33260.83 MB 2025-02-15 03:24:49,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8260.68 MB 2025-02-15 03:24:49,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30299.51 MB 2025-02-15 03:24:50,059 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7827] 2025-02-15 03:24:50,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:50,061 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:24:50,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:50,062 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:24:50,066 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:24:50,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:24:50,068 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:24:50,068 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2 ('] 2025-02-15 03:25:31,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:31,499 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:25:31,504 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:25:31,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:31,508 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:25:31,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:31,509 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:25:33,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:25:33,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:25:33,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.46 seconds 2025-02-15 03:25:33,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:33,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19267.26 MB 2025-02-15 03:25:33,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19829.95 MB 2025-02-15 03:25:33,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-15 03:25:33,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45650.80 MB 2025-02-15 03:25:33,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:33,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20851.98 MB 2025-02-15 03:25:33,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28738.63 MB 2025-02-15 03:25:33,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:25:33,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:25:33,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:25:33,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:33,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19829.95 MB 2025-02-15 03:25:33,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19891.89 MB 2025-02-15 03:25:33,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 61.93 MB 2025-02-15 03:25:33,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:33,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:33,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:33,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21642.42 MB 2025-02-15 03:25:34,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:25:34,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:25:34,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.62 seconds 2025-02-15 03:25:34,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19891.89 MB 2025-02-15 03:25:34,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20063.08 MB 2025-02-15 03:25:34,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 171.20 MB 2025-02-15 03:25:34,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:34,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:34,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24062.57 MB 2025-02-15 03:25:34,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:25:34,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:25:34,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:25:34,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20063.02 MB 2025-02-15 03:25:34,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20672.24 MB 2025-02-15 03:25:34,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 609.23 MB 2025-02-15 03:25:34,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:34,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:34,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21129.37 MB 2025-02-15 03:25:34,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:25:34,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:25:34,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:25:34,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20672.24 MB 2025-02-15 03:25:34,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21395.29 MB 2025-02-15 03:25:34,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 723.04 MB 2025-02-15 03:25:34,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:34,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:34,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23183.27 MB 2025-02-15 03:25:34,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:25:34,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:25:34,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:25:34,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20063.02 MB 2025-02-15 03:25:34,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21395.29 MB 2025-02-15 03:25:34,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1332.27 MB 2025-02-15 03:25:34,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:34,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24798.82 MB 2025-02-15 03:25:34,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23183.27 MB 2025-02-15 03:25:34,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:25:34,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:25:34,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:25:34,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21889.85 MB 2025-02-15 03:25:34,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22137.21 MB 2025-02-15 03:25:34,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 247.36 MB 2025-02-15 03:25:34,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24798.82 MB 2025-02-15 03:25:34,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24926.75 MB 2025-02-15 03:25:34,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 127.93 MB 2025-02-15 03:25:34,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22375.87 MB 2025-02-15 03:25:34,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:25:34,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:25:34,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:25:34,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22270.38 MB 2025-02-15 03:25:34,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22474.32 MB 2025-02-15 03:25:34,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.94 MB 2025-02-15 03:25:34,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24926.75 MB 2025-02-15 03:25:34,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24926.75 MB 2025-02-15 03:25:34,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22474.32 MB 2025-02-15 03:25:34,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:25:34,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:25:34,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-15 03:25:34,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18713.29 MB 2025-02-15 03:25:34,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22653.90 MB 2025-02-15 03:25:34,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3940.61 MB 2025-02-15 03:25:34,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45650.80 MB 2025-02-15 03:25:34,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24926.75 MB 2025-02-15 03:25:34,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20724.06 MB 2025-02-15 03:25:34,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22653.90 MB 2025-02-15 03:25:34,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:25:34,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:25:34,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 03:25:34,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:34,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22653.90 MB 2025-02-15 03:25:34,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22105.80 MB 2025-02-15 03:25:34,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -548.09 MB 2025-02-15 03:25:34,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24926.75 MB 2025-02-15 03:25:34,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24926.75 MB 2025-02-15 03:25:34,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:34,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23910.08 MB 2025-02-15 03:25:35,001 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7288, cut from 7290 2025-02-15 03:25:35,002 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:25:35,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:25:35,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:25:35,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:25:35,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:35,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22105.80 MB 2025-02-15 03:25:35,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29642.36 MB 2025-02-15 03:25:35,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7536.55 MB 2025-02-15 03:25:35,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24926.75 MB 2025-02-15 03:25:35,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34296.82 MB 2025-02-15 03:25:35,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9370.08 MB 2025-02-15 03:25:35,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29642.36 MB 2025-02-15 03:25:35,148 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7080] 2025-02-15 03:25:35,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:35,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:25:35,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:35,150 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:25:35,154 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:25:35,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:35,156 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:25:35,156 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:25:43,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:43,864 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:25:43,869 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:25:43,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:43,872 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 882, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:25:43,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:25:43,873 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 882, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:25:57,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:25:57,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:25:57,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.60 seconds 2025-02-15 03:25:57,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:57,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24305.24 MB 2025-02-15 03:25:57,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27426.59 MB 2025-02-15 03:25:57,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3121.35 MB 2025-02-15 03:25:57,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41792.05 MB 2025-02-15 03:25:57,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33541.85 MB 2025-02-15 03:25:57,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8250.20 MB 2025-02-15 03:25:57,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36268.03 MB 2025-02-15 03:25:57,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:25:57,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:25:57,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:25:57,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:57,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.59 MB 2025-02-15 03:25:57,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25553.71 MB 2025-02-15 03:25:57,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1872.88 MB 2025-02-15 03:25:57,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33541.85 MB 2025-02-15 03:25:57,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41043.36 MB 2025-02-15 03:25:57,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7501.51 MB 2025-02-15 03:25:57,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37733.97 MB 2025-02-15 03:25:59,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:25:59,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:25:59,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:25:59,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25553.71 MB 2025-02-15 03:25:59,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26084.55 MB 2025-02-15 03:25:59,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:25:59,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41043.36 MB 2025-02-15 03:25:59,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31834.77 MB 2025-02-15 03:25:59,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9208.59 MB 2025-02-15 03:25:59,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30063.10 MB 2025-02-15 03:25:59,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:25:59,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:25:59,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:25:59,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26084.55 MB 2025-02-15 03:25:59,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27974.09 MB 2025-02-15 03:25:59,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:25:59,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31834.77 MB 2025-02-15 03:25:59,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32778.49 MB 2025-02-15 03:25:59,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:25:59,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29391.52 MB 2025-02-15 03:25:59,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:25:59,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:25:59,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:25:59,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27974.09 MB 2025-02-15 03:25:59,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30215.94 MB 2025-02-15 03:25:59,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:25:59,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32778.49 MB 2025-02-15 03:25:59,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 03:25:59,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:25:59,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35760.22 MB 2025-02-15 03:25:59,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:25:59,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:25:59,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:25:59,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26084.55 MB 2025-02-15 03:25:59,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30215.94 MB 2025-02-15 03:25:59,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:25:59,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31834.77 MB 2025-02-15 03:25:59,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 03:25:59,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 03:25:59,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35760.22 MB 2025-02-15 03:25:59,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:25:59,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:25:59,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:25:59,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31749.48 MB 2025-02-15 03:25:59,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32516.49 MB 2025-02-15 03:25:59,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:25:59,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38912.66 MB 2025-02-15 03:25:59,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-15 03:25:59,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:25:59,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33224.27 MB 2025-02-15 03:25:59,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:25:59,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:25:59,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:25:59,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32929.38 MB 2025-02-15 03:25:59,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33156.95 MB 2025-02-15 03:25:59,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.57 MB 2025-02-15 03:25:59,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-15 03:25:59,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-15 03:25:59,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:25:59,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33338.88 MB 2025-02-15 03:25:59,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:25:59,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:25:59,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.98 seconds 2025-02-15 03:25:59,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:25:59,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.28 MB 2025-02-15 03:25:59,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33357.95 MB 2025-02-15 03:25:59,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12125.66 MB 2025-02-15 03:25:59,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41792.05 MB 2025-02-15 03:25:59,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-15 03:25:59,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2466.25 MB 2025-02-15 03:25:59,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33357.95 MB 2025-02-15 03:26:00,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:26:00,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:26:00,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:26:00,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:00,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33357.95 MB 2025-02-15 03:26:00,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26235.53 MB 2025-02-15 03:26:00,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7122.42 MB 2025-02-15 03:26:00,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-15 03:26:00,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-15 03:26:00,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:26:00,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35868.69 MB 2025-02-15 03:26:00,142 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 03:26:00,142 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:26:00,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:26:00,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:26:00,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:26:00,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:00,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26235.53 MB 2025-02-15 03:26:00,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34671.12 MB 2025-02-15 03:26:00,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 03:26:00,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-15 03:26:00,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47714.40 MB 2025-02-15 03:26:00,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 03:26:00,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34671.12 MB 2025-02-15 03:26:00,305 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 03:26:00,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:00,307 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:26:00,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:00,308 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:26:00,312 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:26:00,313 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:00,313 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:26:00,313 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:26:50,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:50,368 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:26:50,373 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:26:50,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:50,377 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 172, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:26:50,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:50,378 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 172, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:26:53,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:26:53,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:26:53,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-15 03:26:53,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19357.85 MB 2025-02-15 03:26:53,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19966.55 MB 2025-02-15 03:26:53,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.70 MB 2025-02-15 03:26:53,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56103.01 MB 2025-02-15 03:26:53,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23957.86 MB 2025-02-15 03:26:53,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32145.15 MB 2025-02-15 03:26:53,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28830.02 MB 2025-02-15 03:26:53,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:26:53,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:26:53,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:26:53,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19966.55 MB 2025-02-15 03:26:53,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20261.46 MB 2025-02-15 03:26:53,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.91 MB 2025-02-15 03:26:53,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23957.86 MB 2025-02-15 03:26:53,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23957.86 MB 2025-02-15 03:26:53,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:26:53,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22382.53 MB 2025-02-15 03:26:53,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:26:53,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:26:53,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 03:26:53,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20261.46 MB 2025-02-15 03:26:53,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20489.72 MB 2025-02-15 03:26:53,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-15 03:26:53,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23957.86 MB 2025-02-15 03:26:53,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23957.86 MB 2025-02-15 03:26:53,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:26:53,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24432.15 MB 2025-02-15 03:26:53,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:26:53,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:26:53,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:26:53,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20489.65 MB 2025-02-15 03:26:53,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21301.96 MB 2025-02-15 03:26:53,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.30 MB 2025-02-15 03:26:53,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23957.86 MB 2025-02-15 03:26:53,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23957.86 MB 2025-02-15 03:26:53,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:26:53,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21911.46 MB 2025-02-15 03:26:53,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:26:53,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:26:53,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 03:26:53,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21301.96 MB 2025-02-15 03:26:53,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22265.99 MB 2025-02-15 03:26:53,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 964.04 MB 2025-02-15 03:26:53,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23957.86 MB 2025-02-15 03:26:53,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26398.95 MB 2025-02-15 03:26:53,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2441.08 MB 2025-02-15 03:26:53,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24655.24 MB 2025-02-15 03:26:53,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:26:53,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:26:53,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:26:53,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:53,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20489.65 MB 2025-02-15 03:26:53,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22265.99 MB 2025-02-15 03:26:53,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1776.34 MB 2025-02-15 03:26:53,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23957.86 MB 2025-02-15 03:26:53,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26398.95 MB 2025-02-15 03:26:53,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2441.08 MB 2025-02-15 03:26:53,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24655.24 MB 2025-02-15 03:26:54,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:26:54,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:26:54,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:26:54,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:54,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22925.42 MB 2025-02-15 03:26:54,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23255.75 MB 2025-02-15 03:26:54,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 330.34 MB 2025-02-15 03:26:54,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26398.95 MB 2025-02-15 03:26:54,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26573.01 MB 2025-02-15 03:26:54,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 03:26:54,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23564.81 MB 2025-02-15 03:26:54,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:26:54,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:26:54,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:26:54,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:54,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23433.30 MB 2025-02-15 03:26:54,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23644.58 MB 2025-02-15 03:26:54,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.28 MB 2025-02-15 03:26:54,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26573.01 MB 2025-02-15 03:26:54,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26575.11 MB 2025-02-15 03:26:54,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 03:26:54,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23670.14 MB 2025-02-15 03:26:54,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:26:54,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:26:54,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.70 seconds 2025-02-15 03:26:54,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:54,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18758.58 MB 2025-02-15 03:26:54,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23845.65 MB 2025-02-15 03:26:54,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5087.07 MB 2025-02-15 03:26:54,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56103.01 MB 2025-02-15 03:26:54,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26575.11 MB 2025-02-15 03:26:54,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29527.90 MB 2025-02-15 03:26:54,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23845.65 MB 2025-02-15 03:26:54,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:26:54,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:26:54,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:26:54,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:54,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23845.65 MB 2025-02-15 03:26:54,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22687.50 MB 2025-02-15 03:26:54,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1158.15 MB 2025-02-15 03:26:54,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26575.11 MB 2025-02-15 03:26:54,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26575.11 MB 2025-02-15 03:26:54,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:26:54,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24080.73 MB 2025-02-15 03:26:54,367 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:26:54,368 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-15 03:26:54,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:26:54,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:26:54,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:26:54,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:26:54,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22687.50 MB 2025-02-15 03:26:54,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31127.01 MB 2025-02-15 03:26:54,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.51 MB 2025-02-15 03:26:54,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26575.11 MB 2025-02-15 03:26:54,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37065.06 MB 2025-02-15 03:26:54,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 03:26:54,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31127.01 MB 2025-02-15 03:26:54,535 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:26:54,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:54,537 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:26:54,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:54,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:26:54,542 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:26:54,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:26:54,544 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:26:54,544 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-15 03:28:33,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:33,416 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:28:33,421 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:28:33,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:33,425 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1126, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:28:33,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:33,426 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1126, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:28:50,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:28:50,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:28:50,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.16 seconds 2025-02-15 03:28:50,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:50,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26005.47 MB 2025-02-15 03:28:50,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29990.32 MB 2025-02-15 03:28:50,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3984.85 MB 2025-02-15 03:28:50,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49650.07 MB 2025-02-15 03:28:50,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 03:28:50,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14508.10 MB 2025-02-15 03:28:50,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38874.23 MB 2025-02-15 03:28:50,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:28:50,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:28:50,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:28:50,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:50,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29990.32 MB 2025-02-15 03:28:50,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26823.24 MB 2025-02-15 03:28:50,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3167.08 MB 2025-02-15 03:28:50,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 03:28:50,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44558.19 MB 2025-02-15 03:28:50,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9416.21 MB 2025-02-15 03:28:50,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41507.95 MB 2025-02-15 03:28:52,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:28:52,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:28:52,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 03:28:52,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26823.24 MB 2025-02-15 03:28:52,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27354.08 MB 2025-02-15 03:28:52,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:28:52,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44558.19 MB 2025-02-15 03:28:52,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33279.71 MB 2025-02-15 03:28:52,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11278.48 MB 2025-02-15 03:28:52,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31332.63 MB 2025-02-15 03:28:52,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:28:52,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:28:52,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:28:52,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.08 MB 2025-02-15 03:28:52,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29243.61 MB 2025-02-15 03:28:52,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:28:52,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33279.71 MB 2025-02-15 03:28:52,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-15 03:28:52,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:28:52,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30661.04 MB 2025-02-15 03:28:52,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:28:52,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:28:52,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:28:52,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29243.61 MB 2025-02-15 03:28:52,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31485.47 MB 2025-02-15 03:28:52,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:28:52,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-15 03:28:52,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39885.73 MB 2025-02-15 03:28:52,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:28:52,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37029.75 MB 2025-02-15 03:28:52,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:28:52,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:28:52,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:28:52,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.08 MB 2025-02-15 03:28:52,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31485.47 MB 2025-02-15 03:28:52,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:28:52,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33279.71 MB 2025-02-15 03:28:52,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39885.73 MB 2025-02-15 03:28:52,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:28:52,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37029.75 MB 2025-02-15 03:28:52,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:28:52,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:28:52,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:28:52,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33019.01 MB 2025-02-15 03:28:52,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33786.01 MB 2025-02-15 03:28:52,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:28:52,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39885.73 MB 2025-02-15 03:28:52,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40298.87 MB 2025-02-15 03:28:52,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:28:52,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34493.80 MB 2025-02-15 03:28:52,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:28:52,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:28:52,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:28:52,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34198.90 MB 2025-02-15 03:28:52,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34426.96 MB 2025-02-15 03:28:52,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-15 03:28:52,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40298.87 MB 2025-02-15 03:28:52,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40298.87 MB 2025-02-15 03:28:52,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:28:52,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34622.00 MB 2025-02-15 03:28:52,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:28:52,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:28:52,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.54 seconds 2025-02-15 03:28:52,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:52,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22082.40 MB 2025-02-15 03:28:52,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34627.81 MB 2025-02-15 03:28:52,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12545.41 MB 2025-02-15 03:28:52,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49650.07 MB 2025-02-15 03:28:52,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40298.87 MB 2025-02-15 03:28:52,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9351.20 MB 2025-02-15 03:28:52,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34627.81 MB 2025-02-15 03:28:53,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:28:53,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:28:53,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:28:53,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:53,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34627.81 MB 2025-02-15 03:28:53,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27079.44 MB 2025-02-15 03:28:53,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7548.37 MB 2025-02-15 03:28:53,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40298.87 MB 2025-02-15 03:28:53,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40298.87 MB 2025-02-15 03:28:53,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:28:53,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37133.33 MB 2025-02-15 03:28:53,256 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 03:28:53,256 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:28:53,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:28:53,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:28:53,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:28:53,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:28:53,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27079.44 MB 2025-02-15 03:28:53,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35497.59 MB 2025-02-15 03:28:53,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 03:28:53,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40298.87 MB 2025-02-15 03:28:53,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48668.61 MB 2025-02-15 03:28:53,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-15 03:28:53,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35497.59 MB 2025-02-15 03:28:53,421 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 03:28:53,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:53,422 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:28:53,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:53,423 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:28:53,428 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:28:53,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:28:53,429 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:28:53,429 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:29:09,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:09,655 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:29:09,663 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:29:09,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:09,671 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1968, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:29:09,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:09,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1968, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:29:40,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:29:40,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:29:40,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.45 seconds 2025-02-15 03:29:40,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:40,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31872.66 MB 2025-02-15 03:29:40,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38837.30 MB 2025-02-15 03:29:40,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6964.64 MB 2025-02-15 03:29:40,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61222.16 MB 2025-02-15 03:29:40,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44377.83 MB 2025-02-15 03:29:40,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16844.32 MB 2025-02-15 03:29:40,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47686.63 MB 2025-02-15 03:29:40,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:29:40,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:29:40,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:29:40,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:40,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38837.30 MB 2025-02-15 03:29:40,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31199.49 MB 2025-02-15 03:29:40,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7637.82 MB 2025-02-15 03:29:40,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44377.83 MB 2025-02-15 03:29:40,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59011.76 MB 2025-02-15 03:29:40,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14633.93 MB 2025-02-15 03:29:40,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58717.24 MB 2025-02-15 03:29:42,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:29:42,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:29:42,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:29:42,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31199.49 MB 2025-02-15 03:29:42,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31730.33 MB 2025-02-15 03:29:42,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:29:42,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59011.76 MB 2025-02-15 03:29:42,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-15 03:29:42,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24366.81 MB 2025-02-15 03:29:42,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35708.87 MB 2025-02-15 03:29:42,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:29:42,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:29:42,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:29:42,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31730.33 MB 2025-02-15 03:29:42,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33619.86 MB 2025-02-15 03:29:42,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:29:42,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-15 03:29:42,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37476.11 MB 2025-02-15 03:29:42,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:29:42,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35037.29 MB 2025-02-15 03:29:42,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:29:42,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:29:42,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:29:42,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33619.86 MB 2025-02-15 03:29:42,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35861.72 MB 2025-02-15 03:29:42,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:29:42,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37476.11 MB 2025-02-15 03:29:42,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43610.28 MB 2025-02-15 03:29:42,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:29:42,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41406.00 MB 2025-02-15 03:29:42,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:29:42,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:29:42,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:29:42,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31730.33 MB 2025-02-15 03:29:42,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35861.72 MB 2025-02-15 03:29:42,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:29:42,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-15 03:29:42,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43610.28 MB 2025-02-15 03:29:42,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 03:29:42,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41406.00 MB 2025-02-15 03:29:42,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:29:42,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:29:42,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:29:42,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37395.26 MB 2025-02-15 03:29:42,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38162.26 MB 2025-02-15 03:29:42,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:29:42,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43610.28 MB 2025-02-15 03:29:42,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44023.41 MB 2025-02-15 03:29:42,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:29:42,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38870.05 MB 2025-02-15 03:29:42,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:29:42,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:29:42,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:29:42,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38575.15 MB 2025-02-15 03:29:42,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38804.43 MB 2025-02-15 03:29:42,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.28 MB 2025-02-15 03:29:42,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44023.41 MB 2025-02-15 03:29:42,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44023.41 MB 2025-02-15 03:29:42,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:29:42,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39033.71 MB 2025-02-15 03:29:42,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:29:42,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:29:42,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.95 seconds 2025-02-15 03:29:42,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25015.99 MB 2025-02-15 03:29:42,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39005.31 MB 2025-02-15 03:29:42,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13989.31 MB 2025-02-15 03:29:42,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61222.16 MB 2025-02-15 03:29:42,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44023.41 MB 2025-02-15 03:29:42,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17198.74 MB 2025-02-15 03:29:42,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39033.71 MB 2025-02-15 03:29:42,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:29:42,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:29:42,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:29:42,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39005.31 MB 2025-02-15 03:29:42,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30017.33 MB 2025-02-15 03:29:42,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8987.97 MB 2025-02-15 03:29:42,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44023.41 MB 2025-02-15 03:29:42,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44023.41 MB 2025-02-15 03:29:42,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:29:42,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41514.52 MB 2025-02-15 03:29:42,919 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 03:29:42,919 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:29:42,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:29:42,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:29:42,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:29:42,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:42,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30017.33 MB 2025-02-15 03:29:42,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38448.01 MB 2025-02-15 03:29:42,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 03:29:42,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44023.41 MB 2025-02-15 03:29:42,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52405.73 MB 2025-02-15 03:29:42,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 03:29:42,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38448.01 MB 2025-02-15 03:29:43,082 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 03:29:43,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:43,083 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:29:43,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:43,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:29:43,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:29:43,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:43,090 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:29:43,090 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:29:51,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:51,622 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:29:51,627 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:29:51,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:51,630 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:29:51,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:51,631 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:29:56,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:29:56,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:29:56,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.76 seconds 2025-02-15 03:29:56,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:56,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20284.61 MB 2025-02-15 03:29:56,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21363.99 MB 2025-02-15 03:29:56,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1079.38 MB 2025-02-15 03:29:56,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64978.16 MB 2025-02-15 03:29:56,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27684.50 MB 2025-02-15 03:29:56,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37293.65 MB 2025-02-15 03:29:56,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30208.97 MB 2025-02-15 03:29:56,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:29:56,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:29:56,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:29:56,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:56,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21363.99 MB 2025-02-15 03:29:56,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21803.54 MB 2025-02-15 03:29:56,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.55 MB 2025-02-15 03:29:56,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27684.50 MB 2025-02-15 03:29:56,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29225.91 MB 2025-02-15 03:29:56,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1541.41 MB 2025-02-15 03:29:56,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25480.42 MB 2025-02-15 03:29:57,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:29:57,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:29:57,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.41 seconds 2025-02-15 03:29:57,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:57,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21803.54 MB 2025-02-15 03:29:57,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22192.38 MB 2025-02-15 03:29:57,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 388.84 MB 2025-02-15 03:29:57,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29225.91 MB 2025-02-15 03:29:57,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28282.19 MB 2025-02-15 03:29:57,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 03:29:57,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26143.06 MB 2025-02-15 03:29:57,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:29:57,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:29:57,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:29:57,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:57,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22192.38 MB 2025-02-15 03:29:57,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23577.03 MB 2025-02-15 03:29:57,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1384.64 MB 2025-02-15 03:29:57,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28282.19 MB 2025-02-15 03:29:57,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28282.19 MB 2025-02-15 03:29:57,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:29:57,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24615.30 MB 2025-02-15 03:29:57,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:29:57,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:29:57,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:29:57,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:57,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23577.03 MB 2025-02-15 03:29:57,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25219.20 MB 2025-02-15 03:29:57,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.18 MB 2025-02-15 03:29:57,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28282.19 MB 2025-02-15 03:29:57,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-15 03:29:57,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3806.33 MB 2025-02-15 03:29:57,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29283.91 MB 2025-02-15 03:29:57,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:29:57,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:29:57,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:29:57,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:57,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22192.38 MB 2025-02-15 03:29:57,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25219.20 MB 2025-02-15 03:29:57,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3026.82 MB 2025-02-15 03:29:57,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28282.19 MB 2025-02-15 03:29:57,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-15 03:29:57,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3806.33 MB 2025-02-15 03:29:57,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29283.91 MB 2025-02-15 03:29:58,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:29:58,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:29:58,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:29:58,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:58,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26342.52 MB 2025-02-15 03:29:58,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26904.74 MB 2025-02-15 03:29:58,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.22 MB 2025-02-15 03:29:58,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-15 03:29:58,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 03:29:58,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 297.80 MB 2025-02-15 03:29:58,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27423.20 MB 2025-02-15 03:29:58,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:29:58,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:29:58,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:29:58,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:58,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27207.19 MB 2025-02-15 03:29:58,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27425.52 MB 2025-02-15 03:29:58,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.33 MB 2025-02-15 03:29:58,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 03:29:58,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 03:29:58,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:29:58,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27511.25 MB 2025-02-15 03:29:58,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:29:58,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:29:58,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.50 seconds 2025-02-15 03:29:58,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:58,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19221.97 MB 2025-02-15 03:29:58,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27626.59 MB 2025-02-15 03:29:58,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.62 MB 2025-02-15 03:29:58,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64978.16 MB 2025-02-15 03:29:58,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 03:29:58,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32591.84 MB 2025-02-15 03:29:58,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27626.59 MB 2025-02-15 03:29:58,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:29:58,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:29:58,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:29:58,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:58,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27626.59 MB 2025-02-15 03:29:58,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30640.62 MB 2025-02-15 03:29:58,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 03:29:58,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 03:29:58,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 03:29:58,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:29:58,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30941.99 MB 2025-02-15 03:29:58,417 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:29:58,417 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:29:58,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:29:58,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:29:58,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:29:58,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:29:58,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23721.79 MB 2025-02-15 03:29:58,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32160.81 MB 2025-02-15 03:29:58,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:29:58,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 03:29:58,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42876.27 MB 2025-02-15 03:29:58,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 03:29:58,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32160.81 MB 2025-02-15 03:29:58,582 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:29:58,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:58,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:29:58,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:58,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:29:58,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:29:58,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:29:58,590 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:29:58,590 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:31:14,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:14,023 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:31:14,030 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:31:14,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:14,037 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 112, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:31:14,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:14,038 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 112, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:31:15,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:31:15,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:31:15,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.78 seconds 2025-02-15 03:31:15,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:15,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18939.76 MB 2025-02-15 03:31:15,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19336.12 MB 2025-02-15 03:31:15,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 396.36 MB 2025-02-15 03:31:15,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55461.28 MB 2025-02-15 03:31:15,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-15 03:31:15,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34445.72 MB 2025-02-15 03:31:15,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28184.64 MB 2025-02-15 03:31:15,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:31:15,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:31:15,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:31:15,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:15,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19336.12 MB 2025-02-15 03:31:15,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19528.16 MB 2025-02-15 03:31:15,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.04 MB 2025-02-15 03:31:15,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-15 03:31:15,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21411.92 MB 2025-02-15 03:31:15,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 396.36 MB 2025-02-15 03:31:15,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20122.76 MB 2025-02-15 03:31:16,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:31:16,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:31:16,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.56 seconds 2025-02-15 03:31:16,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19528.16 MB 2025-02-15 03:31:16,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19676.79 MB 2025-02-15 03:31:16,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 148.64 MB 2025-02-15 03:31:16,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21411.92 MB 2025-02-15 03:31:16,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21411.92 MB 2025-02-15 03:31:16,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:31:16,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23614.95 MB 2025-02-15 03:31:16,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:31:16,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:31:16,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:31:16,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19676.73 MB 2025-02-15 03:31:16,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20205.67 MB 2025-02-15 03:31:16,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 528.94 MB 2025-02-15 03:31:16,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21411.92 MB 2025-02-15 03:31:16,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-15 03:31:16,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 264.24 MB 2025-02-15 03:31:16,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20602.55 MB 2025-02-15 03:31:16,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:31:16,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:31:16,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 03:31:16,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20205.67 MB 2025-02-15 03:31:16,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20848.11 MB 2025-02-15 03:31:16,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 642.45 MB 2025-02-15 03:31:16,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-15 03:31:16,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23261.61 MB 2025-02-15 03:31:16,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1585.45 MB 2025-02-15 03:31:16,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22385.78 MB 2025-02-15 03:31:16,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:31:16,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:31:16,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 03:31:16,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19676.73 MB 2025-02-15 03:31:16,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20848.11 MB 2025-02-15 03:31:16,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.39 MB 2025-02-15 03:31:16,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21411.92 MB 2025-02-15 03:31:16,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23261.61 MB 2025-02-15 03:31:16,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1849.69 MB 2025-02-15 03:31:16,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22385.78 MB 2025-02-15 03:31:16,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:31:16,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:31:16,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 03:31:16,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21468.34 MB 2025-02-15 03:31:16,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21738.16 MB 2025-02-15 03:31:16,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.81 MB 2025-02-15 03:31:16,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23261.61 MB 2025-02-15 03:31:16,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23427.28 MB 2025-02-15 03:31:16,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 03:31:16,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21936.34 MB 2025-02-15 03:31:16,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:31:16,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:31:16,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:31:16,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.82 MB 2025-02-15 03:31:16,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22138.50 MB 2025-02-15 03:31:16,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.67 MB 2025-02-15 03:31:16,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23427.28 MB 2025-02-15 03:31:16,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23427.28 MB 2025-02-15 03:31:16,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:31:16,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22138.50 MB 2025-02-15 03:31:16,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:31:16,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:31:16,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.61 seconds 2025-02-15 03:31:16,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18549.54 MB 2025-02-15 03:31:16,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22338.91 MB 2025-02-15 03:31:16,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3789.37 MB 2025-02-15 03:31:16,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55461.28 MB 2025-02-15 03:31:16,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23427.28 MB 2025-02-15 03:31:16,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32034.00 MB 2025-02-15 03:31:16,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22338.91 MB 2025-02-15 03:31:16,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:31:16,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:31:16,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 03:31:16,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22338.91 MB 2025-02-15 03:31:16,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25342.99 MB 2025-02-15 03:31:16,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.08 MB 2025-02-15 03:31:16,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23427.28 MB 2025-02-15 03:31:16,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27185.38 MB 2025-02-15 03:31:16,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3758.10 MB 2025-02-15 03:31:16,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25644.06 MB 2025-02-15 03:31:16,965 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-15 03:31:16,966 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:31:16,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:31:16,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:31:16,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:31:16,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:31:16,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25342.99 MB 2025-02-15 03:31:16,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33753.81 MB 2025-02-15 03:31:16,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-15 03:31:16,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27185.38 MB 2025-02-15 03:31:16,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37639.68 MB 2025-02-15 03:31:16,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-15 03:31:16,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33753.81 MB 2025-02-15 03:31:17,233 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-15 03:31:17,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:17,235 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:31:17,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:17,238 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:31:17,246 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:31:17,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:31:17,248 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:31:17,248 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 03:32:58,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:32:58,073 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:32:58,081 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:32:58,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:32:58,088 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:32:58,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:32:58,089 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:33:23,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:33:23,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:33:23,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.56 seconds 2025-02-15 03:33:23,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:23,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29768.28 MB 2025-02-15 03:33:23,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35664.16 MB 2025-02-15 03:33:23,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5895.88 MB 2025-02-15 03:33:23,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46005.22 MB 2025-02-15 03:33:23,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43299.90 MB 2025-02-15 03:33:23,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2705.33 MB 2025-02-15 03:33:23,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44676.27 MB 2025-02-15 03:33:23,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:33:23,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:33:23,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:33:23,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:23,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35664.16 MB 2025-02-15 03:33:23,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29629.48 MB 2025-02-15 03:33:23,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6034.68 MB 2025-02-15 03:33:23,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43299.90 MB 2025-02-15 03:33:23,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55176.07 MB 2025-02-15 03:33:23,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11876.17 MB 2025-02-15 03:33:23,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51393.17 MB 2025-02-15 03:33:25,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:33:25,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:33:25,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 03:33:25,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:25,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29629.48 MB 2025-02-15 03:33:25,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30160.32 MB 2025-02-15 03:33:25,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:33:25,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55176.07 MB 2025-02-15 03:33:25,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 03:33:25,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16357.79 MB 2025-02-15 03:33:25,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34138.87 MB 2025-02-15 03:33:25,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:33:25,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:33:25,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:33:25,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:25,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30160.32 MB 2025-02-15 03:33:25,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32049.86 MB 2025-02-15 03:33:25,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:33:25,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 03:33:25,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 03:33:25,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:33:25,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33467.29 MB 2025-02-15 03:33:25,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:33:25,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:33:25,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:33:25,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:25,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32049.86 MB 2025-02-15 03:33:25,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34291.71 MB 2025-02-15 03:33:25,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:33:25,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 03:33:25,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43065.02 MB 2025-02-15 03:33:25,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 03:33:25,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39835.99 MB 2025-02-15 03:33:25,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:33:25,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:33:25,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:33:25,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:25,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30160.32 MB 2025-02-15 03:33:25,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34291.71 MB 2025-02-15 03:33:25,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:33:25,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 03:33:25,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43065.02 MB 2025-02-15 03:33:25,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 03:33:25,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39835.99 MB 2025-02-15 03:33:26,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:33:26,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:33:26,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:33:26,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:26,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35825.26 MB 2025-02-15 03:33:26,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36592.26 MB 2025-02-15 03:33:26,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:33:26,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43065.02 MB 2025-02-15 03:33:26,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43471.86 MB 2025-02-15 03:33:26,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 406.85 MB 2025-02-15 03:33:26,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37300.05 MB 2025-02-15 03:33:26,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:33:26,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:33:26,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:33:26,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:26,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37005.15 MB 2025-02-15 03:33:26,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37234.18 MB 2025-02-15 03:33:26,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-15 03:33:26,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43471.86 MB 2025-02-15 03:33:26,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43471.86 MB 2025-02-15 03:33:26,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:33:26,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37444.83 MB 2025-02-15 03:33:26,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:33:26,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:33:26,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.99 seconds 2025-02-15 03:33:26,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:26,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23963.80 MB 2025-02-15 03:33:26,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37435.13 MB 2025-02-15 03:33:26,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13471.33 MB 2025-02-15 03:33:26,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46005.22 MB 2025-02-15 03:33:26,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43471.86 MB 2025-02-15 03:33:26,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2533.36 MB 2025-02-15 03:33:26,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37444.83 MB 2025-02-15 03:33:26,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:33:26,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:33:26,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:33:26,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:26,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37435.13 MB 2025-02-15 03:33:26,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.28 MB 2025-02-15 03:33:26,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8468.85 MB 2025-02-15 03:33:26,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43471.86 MB 2025-02-15 03:33:26,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43471.86 MB 2025-02-15 03:33:26,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:33:26,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39945.26 MB 2025-02-15 03:33:26,367 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 03:33:26,368 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:33:26,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:33:26,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:33:26,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:33:26,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:33:26,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28966.28 MB 2025-02-15 03:33:26,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37400.90 MB 2025-02-15 03:33:26,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 03:33:26,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43471.86 MB 2025-02-15 03:33:26,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51856.28 MB 2025-02-15 03:33:26,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 03:33:26,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37400.90 MB 2025-02-15 03:33:26,534 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 03:33:26,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:33:26,535 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:33:26,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:33:26,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:33:26,541 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:33:26,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:33:26,542 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:33:26,542 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:34:30,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:34:30,609 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:34:30,614 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:34:30,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:34:30,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:34:30,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:34:30,619 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:35:05,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:35:05,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:35:05,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.37 seconds 2025-02-15 03:35:05,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:05,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33663.48 MB 2025-02-15 03:35:05,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41538.29 MB 2025-02-15 03:35:05,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.81 MB 2025-02-15 03:35:05,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60240.69 MB 2025-02-15 03:35:05,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45281.71 MB 2025-02-15 03:35:05,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14958.99 MB 2025-02-15 03:35:05,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50383.41 MB 2025-02-15 03:35:05,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:35:05,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:35:05,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:35:05,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:05,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41538.29 MB 2025-02-15 03:35:05,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32536.60 MB 2025-02-15 03:35:05,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9001.69 MB 2025-02-15 03:35:05,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45281.71 MB 2025-02-15 03:35:05,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62402.85 MB 2025-02-15 03:35:05,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17121.15 MB 2025-02-15 03:35:05,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64090.99 MB 2025-02-15 03:35:07,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:35:07,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:35:07,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 03:35:07,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32536.60 MB 2025-02-15 03:35:07,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33067.44 MB 2025-02-15 03:35:07,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:35:07,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62402.85 MB 2025-02-15 03:35:07,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35882.27 MB 2025-02-15 03:35:07,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26520.58 MB 2025-02-15 03:35:07,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37045.99 MB 2025-02-15 03:35:07,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:35:07,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:35:07,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:35:07,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33067.44 MB 2025-02-15 03:35:07,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34956.71 MB 2025-02-15 03:35:07,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 03:35:07,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35882.27 MB 2025-02-15 03:35:07,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39657.14 MB 2025-02-15 03:35:07,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 03:35:07,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36374.14 MB 2025-02-15 03:35:07,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:35:07,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:35:07,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:35:07,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34956.71 MB 2025-02-15 03:35:07,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37198.57 MB 2025-02-15 03:35:07,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:35:07,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39657.14 MB 2025-02-15 03:35:07,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-15 03:35:07,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:35:07,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42742.85 MB 2025-02-15 03:35:07,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:35:07,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:35:07,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:35:07,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33067.44 MB 2025-02-15 03:35:07,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37198.57 MB 2025-02-15 03:35:07,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 03:35:07,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35882.27 MB 2025-02-15 03:35:07,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-15 03:35:07,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 03:35:07,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42742.85 MB 2025-02-15 03:35:07,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:35:07,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:35:07,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 03:35:07,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38732.11 MB 2025-02-15 03:35:07,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39499.11 MB 2025-02-15 03:35:07,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:35:07,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45791.31 MB 2025-02-15 03:35:07,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46202.36 MB 2025-02-15 03:35:07,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 03:35:07,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40206.90 MB 2025-02-15 03:35:07,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:35:07,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:35:07,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:35:07,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39912.00 MB 2025-02-15 03:35:07,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40141.24 MB 2025-02-15 03:35:07,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.24 MB 2025-02-15 03:35:07,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46202.36 MB 2025-02-15 03:35:07,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46202.36 MB 2025-02-15 03:35:07,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:35:07,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40368.15 MB 2025-02-15 03:35:07,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:35:07,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:35:07,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.96 seconds 2025-02-15 03:35:07,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25911.40 MB 2025-02-15 03:35:07,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40341.38 MB 2025-02-15 03:35:07,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14429.98 MB 2025-02-15 03:35:07,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60240.69 MB 2025-02-15 03:35:07,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46202.36 MB 2025-02-15 03:35:07,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14038.34 MB 2025-02-15 03:35:07,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40368.15 MB 2025-02-15 03:35:07,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:35:07,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:35:07,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:35:07,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40341.38 MB 2025-02-15 03:35:07,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30902.21 MB 2025-02-15 03:35:07,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9439.17 MB 2025-02-15 03:35:07,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46202.36 MB 2025-02-15 03:35:07,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46202.36 MB 2025-02-15 03:35:07,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:35:07,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42841.38 MB 2025-02-15 03:35:07,871 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 03:35:07,871 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:35:07,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:35:07,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:35:07,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:35:07,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:35:07,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30902.21 MB 2025-02-15 03:35:07,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39303.07 MB 2025-02-15 03:35:07,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 03:35:07,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46202.36 MB 2025-02-15 03:35:07,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54553.21 MB 2025-02-15 03:35:07,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 03:35:07,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39303.07 MB 2025-02-15 03:35:08,037 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 03:35:08,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:35:08,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:35:08,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:35:08,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:35:08,044 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:35:08,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:35:08,045 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:35:08,045 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:36:00,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:00,664 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:36:00,670 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:36:00,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:00,674 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1334, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:36:00,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:00,675 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1334, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:36:21,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:36:21,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:36:21,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.59 seconds 2025-02-15 03:36:21,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:21,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27454.85 MB 2025-02-15 03:36:21,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32175.80 MB 2025-02-15 03:36:21,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4720.95 MB 2025-02-15 03:36:21,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62904.07 MB 2025-02-15 03:36:21,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42094.03 MB 2025-02-15 03:36:21,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20810.04 MB 2025-02-15 03:36:21,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41003.08 MB 2025-02-15 03:36:21,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:36:21,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:36:21,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:36:21,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:21,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32175.80 MB 2025-02-15 03:36:21,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27903.52 MB 2025-02-15 03:36:21,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4272.28 MB 2025-02-15 03:36:21,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42094.03 MB 2025-02-15 03:36:21,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51430.56 MB 2025-02-15 03:36:21,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9336.52 MB 2025-02-15 03:36:21,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46214.24 MB 2025-02-15 03:36:23,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:36:23,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:36:23,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:36:23,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27903.52 MB 2025-02-15 03:36:23,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28434.36 MB 2025-02-15 03:36:23,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:36:23,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51430.56 MB 2025-02-15 03:36:23,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33195.82 MB 2025-02-15 03:36:23,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18234.74 MB 2025-02-15 03:36:23,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32412.91 MB 2025-02-15 03:36:23,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:36:23,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:36:23,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:36:23,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28434.36 MB 2025-02-15 03:36:23,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30324.60 MB 2025-02-15 03:36:23,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1890.24 MB 2025-02-15 03:36:23,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33195.82 MB 2025-02-15 03:36:23,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35083.26 MB 2025-02-15 03:36:23,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 03:36:23,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31742.03 MB 2025-02-15 03:36:23,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:36:23,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:36:23,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:36:23,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30324.60 MB 2025-02-15 03:36:23,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32566.45 MB 2025-02-15 03:36:23,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:36:23,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35083.26 MB 2025-02-15 03:36:23,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40745.57 MB 2025-02-15 03:36:23,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:36:23,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38111.44 MB 2025-02-15 03:36:23,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:36:23,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:36:23,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:36:23,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28434.36 MB 2025-02-15 03:36:23,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32566.45 MB 2025-02-15 03:36:23,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.10 MB 2025-02-15 03:36:23,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33195.82 MB 2025-02-15 03:36:23,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40745.57 MB 2025-02-15 03:36:23,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 03:36:23,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38111.44 MB 2025-02-15 03:36:23,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:36:23,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:36:23,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:36:23,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34100.00 MB 2025-02-15 03:36:23,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34867.70 MB 2025-02-15 03:36:23,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.71 MB 2025-02-15 03:36:23,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40745.57 MB 2025-02-15 03:36:23,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41156.61 MB 2025-02-15 03:36:23,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 03:36:23,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35575.49 MB 2025-02-15 03:36:23,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:36:23,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:36:23,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:36:23,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35280.59 MB 2025-02-15 03:36:23,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35508.07 MB 2025-02-15 03:36:23,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.47 MB 2025-02-15 03:36:23,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41156.61 MB 2025-02-15 03:36:23,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41156.61 MB 2025-02-15 03:36:23,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:36:23,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35729.18 MB 2025-02-15 03:36:23,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:36:23,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:36:23,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.01 seconds 2025-02-15 03:36:23,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.09 MB 2025-02-15 03:36:23,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35708.92 MB 2025-02-15 03:36:23,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12901.83 MB 2025-02-15 03:36:23,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62904.07 MB 2025-02-15 03:36:23,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41156.61 MB 2025-02-15 03:36:23,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21747.47 MB 2025-02-15 03:36:23,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35729.18 MB 2025-02-15 03:36:23,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:36:23,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:36:23,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:36:23,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35708.92 MB 2025-02-15 03:36:23,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27796.19 MB 2025-02-15 03:36:23,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7912.72 MB 2025-02-15 03:36:23,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41156.61 MB 2025-02-15 03:36:23,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41156.61 MB 2025-02-15 03:36:23,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:36:23,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38206.76 MB 2025-02-15 03:36:23,976 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 03:36:23,976 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:36:23,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:36:23,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:36:23,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:36:23,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:36:23,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27796.19 MB 2025-02-15 03:36:23,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36188.44 MB 2025-02-15 03:36:23,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.24 MB 2025-02-15 03:36:23,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41156.61 MB 2025-02-15 03:36:23,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45329.94 MB 2025-02-15 03:36:23,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 03:36:23,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36188.44 MB 2025-02-15 03:36:24,140 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 03:36:24,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:24,142 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:36:24,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:24,143 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:36:24,148 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:36:24,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:36:24,149 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:36:24,149 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:37:03,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:03,149 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:37:03,156 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:37:03,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:03,163 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:37:03,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:03,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:37:21,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:37:21,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:37:21,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.35 seconds 2025-02-15 03:37:21,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:21,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26360.85 MB 2025-02-15 03:37:21,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30526.18 MB 2025-02-15 03:37:21,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4165.34 MB 2025-02-15 03:37:21,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53672.41 MB 2025-02-15 03:37:21,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33176.94 MB 2025-02-15 03:37:21,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20495.47 MB 2025-02-15 03:37:21,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39456.90 MB 2025-02-15 03:37:21,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:37:21,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:37:21,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 03:37:21,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:21,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30526.18 MB 2025-02-15 03:37:21,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27088.37 MB 2025-02-15 03:37:21,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3437.81 MB 2025-02-15 03:37:21,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33176.94 MB 2025-02-15 03:37:21,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43572.53 MB 2025-02-15 03:37:21,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10395.58 MB 2025-02-15 03:37:21,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43018.12 MB 2025-02-15 03:37:23,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:37:23,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:37:23,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:37:23,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:23,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27088.37 MB 2025-02-15 03:37:23,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27619.21 MB 2025-02-15 03:37:23,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:37:23,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43572.53 MB 2025-02-15 03:37:23,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31134.32 MB 2025-02-15 03:37:23,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12438.21 MB 2025-02-15 03:37:23,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31598.80 MB 2025-02-15 03:37:23,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:37:23,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:37:23,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:37:23,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:23,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27619.21 MB 2025-02-15 03:37:23,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29508.75 MB 2025-02-15 03:37:23,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:37:23,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31134.32 MB 2025-02-15 03:37:23,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33965.47 MB 2025-02-15 03:37:23,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:37:23,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30926.18 MB 2025-02-15 03:37:23,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:37:23,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:37:23,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:37:23,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:23,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29508.75 MB 2025-02-15 03:37:23,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31750.60 MB 2025-02-15 03:37:23,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:37:23,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33965.47 MB 2025-02-15 03:37:23,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39627.78 MB 2025-02-15 03:37:23,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:37:23,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37294.89 MB 2025-02-15 03:37:23,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:37:23,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:37:23,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:37:23,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:23,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27619.21 MB 2025-02-15 03:37:23,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31750.60 MB 2025-02-15 03:37:23,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:37:23,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31134.32 MB 2025-02-15 03:37:23,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39627.78 MB 2025-02-15 03:37:23,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 03:37:23,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37294.89 MB 2025-02-15 03:37:24,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:37:24,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:37:24,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:37:24,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:24,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33284.15 MB 2025-02-15 03:37:24,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34051.15 MB 2025-02-15 03:37:24,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:37:24,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39627.78 MB 2025-02-15 03:37:24,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40040.92 MB 2025-02-15 03:37:24,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:37:24,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34758.94 MB 2025-02-15 03:37:24,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:37:24,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:37:24,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:37:24,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:24,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34464.04 MB 2025-02-15 03:37:24,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34692.26 MB 2025-02-15 03:37:24,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 03:37:24,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40040.92 MB 2025-02-15 03:37:24,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40040.92 MB 2025-02-15 03:37:24,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:37:24,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34931.10 MB 2025-02-15 03:37:24,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:37:24,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:37:24,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.88 seconds 2025-02-15 03:37:24,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:24,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22260.09 MB 2025-02-15 03:37:24,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34892.40 MB 2025-02-15 03:37:24,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12632.31 MB 2025-02-15 03:37:24,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53672.41 MB 2025-02-15 03:37:24,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40040.92 MB 2025-02-15 03:37:24,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13631.49 MB 2025-02-15 03:37:24,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34931.10 MB 2025-02-15 03:37:24,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:37:24,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:37:24,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 03:37:24,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:24,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34892.40 MB 2025-02-15 03:37:24,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27250.90 MB 2025-02-15 03:37:24,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7641.50 MB 2025-02-15 03:37:24,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40040.92 MB 2025-02-15 03:37:24,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40040.92 MB 2025-02-15 03:37:24,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:37:24,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37393.11 MB 2025-02-15 03:37:24,363 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 03:37:24,363 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:37:24,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:37:24,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:37:24,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:37:24,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:37:24,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27250.90 MB 2025-02-15 03:37:24,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35651.76 MB 2025-02-15 03:37:24,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 03:37:24,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40040.92 MB 2025-02-15 03:37:24,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48391.78 MB 2025-02-15 03:37:24,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 03:37:24,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35651.76 MB 2025-02-15 03:37:24,622 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 03:37:24,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:24,625 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:37:24,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:24,627 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:37:24,634 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:37:24,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:37:24,636 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:37:24,637 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:38:48,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:38:48,680 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:38:48,685 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:38:48,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:38:48,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 875, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:38:48,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:38:48,690 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 875, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:39:02,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:39:02,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:39:02,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.47 seconds 2025-02-15 03:39:02,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:02,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24256.46 MB 2025-02-15 03:39:02,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27353.96 MB 2025-02-15 03:39:02,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3097.49 MB 2025-02-15 03:39:02,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56742.64 MB 2025-02-15 03:39:02,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32111.59 MB 2025-02-15 03:39:02,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24631.05 MB 2025-02-15 03:39:02,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36220.06 MB 2025-02-15 03:39:02,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:39:02,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:39:02,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:39:02,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:02,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27353.96 MB 2025-02-15 03:39:02,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25518.37 MB 2025-02-15 03:39:02,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1835.59 MB 2025-02-15 03:39:02,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32111.59 MB 2025-02-15 03:39:02,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39472.59 MB 2025-02-15 03:39:02,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7361.00 MB 2025-02-15 03:39:02,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36812.80 MB 2025-02-15 03:39:04,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:39:04,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:39:04,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 03:39:04,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25518.37 MB 2025-02-15 03:39:04,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26049.21 MB 2025-02-15 03:39:04,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:39:04,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39472.59 MB 2025-02-15 03:39:04,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31138.51 MB 2025-02-15 03:39:04,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8334.08 MB 2025-02-15 03:39:04,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30027.76 MB 2025-02-15 03:39:04,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:39:04,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:39:04,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:39:04,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26049.21 MB 2025-02-15 03:39:04,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27938.74 MB 2025-02-15 03:39:04,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:39:04,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31138.51 MB 2025-02-15 03:39:04,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-15 03:39:04,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:39:04,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29356.17 MB 2025-02-15 03:39:04,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:39:04,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:39:04,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.40 seconds 2025-02-15 03:39:04,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27938.74 MB 2025-02-15 03:39:04,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24989.98 MB 2025-02-15 03:39:04,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2948.76 MB 2025-02-15 03:39:04,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-15 03:39:04,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33025.95 MB 2025-02-15 03:39:04,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:39:04,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30534.26 MB 2025-02-15 03:39:04,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:39:04,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:39:04,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-15 03:39:04,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26049.21 MB 2025-02-15 03:39:04,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24989.98 MB 2025-02-15 03:39:04,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1059.23 MB 2025-02-15 03:39:04,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31138.51 MB 2025-02-15 03:39:04,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33025.95 MB 2025-02-15 03:39:04,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 03:39:04,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30534.26 MB 2025-02-15 03:39:04,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:39:04,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:39:04,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 03:39:04,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.53 MB 2025-02-15 03:39:04,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27290.53 MB 2025-02-15 03:39:04,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:39:04,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33025.95 MB 2025-02-15 03:39:04,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33432.80 MB 2025-02-15 03:39:04,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 406.85 MB 2025-02-15 03:39:04,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27998.32 MB 2025-02-15 03:39:04,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:39:04,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:39:04,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:39:04,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27703.42 MB 2025-02-15 03:39:04,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27929.70 MB 2025-02-15 03:39:04,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.28 MB 2025-02-15 03:39:04,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33432.80 MB 2025-02-15 03:39:04,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33432.80 MB 2025-02-15 03:39:04,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:39:04,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28145.21 MB 2025-02-15 03:39:04,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:39:04,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:39:04,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.07 seconds 2025-02-15 03:39:04,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:04,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21207.89 MB 2025-02-15 03:39:04,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28130.77 MB 2025-02-15 03:39:04,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6922.88 MB 2025-02-15 03:39:04,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56742.64 MB 2025-02-15 03:39:04,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33432.80 MB 2025-02-15 03:39:04,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23309.84 MB 2025-02-15 03:39:04,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28145.21 MB 2025-02-15 03:39:05,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:39:05,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:39:05,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:39:05,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:05,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28130.77 MB 2025-02-15 03:39:05,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21021.67 MB 2025-02-15 03:39:05,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7109.11 MB 2025-02-15 03:39:05,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33432.80 MB 2025-02-15 03:39:05,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33432.80 MB 2025-02-15 03:39:05,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:39:05,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28130.77 MB 2025-02-15 03:39:05,046 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:39:05,047 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:39:05,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:39:05,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:39:05,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:39:05,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:39:05,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21021.67 MB 2025-02-15 03:39:05,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29460.69 MB 2025-02-15 03:39:05,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:39:05,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33432.80 MB 2025-02-15 03:39:05,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41823.50 MB 2025-02-15 03:39:05,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:39:05,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.69 MB 2025-02-15 03:39:05,220 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:39:05,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:39:05,222 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:39:05,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:39:05,223 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:39:05,228 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:39:05,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:39:05,229 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:39:05,229 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:40:04,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:04,662 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:40:04,668 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:40:04,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:04,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1933, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:40:04,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:04,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1933, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:40:34,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:40:34,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:40:34,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.76 seconds 2025-02-15 03:40:34,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:34,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26438.16 MB 2025-02-15 03:40:34,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33279.07 MB 2025-02-15 03:40:34,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6840.91 MB 2025-02-15 03:40:34,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54408.51 MB 2025-02-15 03:40:34,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40040.92 MB 2025-02-15 03:40:34,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14367.59 MB 2025-02-15 03:40:34,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42251.32 MB 2025-02-15 03:40:34,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:40:34,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:40:34,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 03:40:34,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:34,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33279.07 MB 2025-02-15 03:40:34,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25826.91 MB 2025-02-15 03:40:34,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7452.15 MB 2025-02-15 03:40:34,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40040.92 MB 2025-02-15 03:40:34,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54305.75 MB 2025-02-15 03:40:34,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14264.83 MB 2025-02-15 03:40:34,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52599.57 MB 2025-02-15 03:40:36,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:40:36,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:40:36,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:40:36,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25826.91 MB 2025-02-15 03:40:36,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26357.76 MB 2025-02-15 03:40:36,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:40:36,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54305.75 MB 2025-02-15 03:40:36,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30444.36 MB 2025-02-15 03:40:36,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23861.40 MB 2025-02-15 03:40:36,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.30 MB 2025-02-15 03:40:36,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:40:36,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:40:36,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:40:36,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26357.76 MB 2025-02-15 03:40:36,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28247.29 MB 2025-02-15 03:40:36,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:40:36,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30444.36 MB 2025-02-15 03:40:36,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32331.79 MB 2025-02-15 03:40:36,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 03:40:36,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29664.72 MB 2025-02-15 03:40:36,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:40:36,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:40:36,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:40:36,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28247.29 MB 2025-02-15 03:40:36,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30489.15 MB 2025-02-15 03:40:36,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:40:36,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32331.79 MB 2025-02-15 03:40:36,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37994.10 MB 2025-02-15 03:40:36,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:40:36,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36033.43 MB 2025-02-15 03:40:36,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:40:36,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:40:36,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:40:36,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26357.76 MB 2025-02-15 03:40:36,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30489.15 MB 2025-02-15 03:40:36,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:40:36,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30444.36 MB 2025-02-15 03:40:36,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37994.10 MB 2025-02-15 03:40:36,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 03:40:36,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36033.43 MB 2025-02-15 03:40:36,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:40:36,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:40:36,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:40:36,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32022.69 MB 2025-02-15 03:40:36,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32789.69 MB 2025-02-15 03:40:36,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:40:36,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37994.10 MB 2025-02-15 03:40:36,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38409.34 MB 2025-02-15 03:40:36,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:40:36,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33497.48 MB 2025-02-15 03:40:36,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:40:36,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:40:36,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:40:36,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33202.58 MB 2025-02-15 03:40:36,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33430.07 MB 2025-02-15 03:40:36,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.49 MB 2025-02-15 03:40:36,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38409.34 MB 2025-02-15 03:40:36,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38409.34 MB 2025-02-15 03:40:36,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:40:36,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33635.81 MB 2025-02-15 03:40:36,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:40:36,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:40:36,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.26 seconds 2025-02-15 03:40:36,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:36,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19703.43 MB 2025-02-15 03:40:36,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33630.62 MB 2025-02-15 03:40:36,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13927.19 MB 2025-02-15 03:40:36,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54408.51 MB 2025-02-15 03:40:36,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38409.34 MB 2025-02-15 03:40:36,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15999.17 MB 2025-02-15 03:40:36,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33635.81 MB 2025-02-15 03:40:37,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:40:37,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:40:37,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:40:37,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:37,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33630.62 MB 2025-02-15 03:40:37,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24700.12 MB 2025-02-15 03:40:37,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8930.50 MB 2025-02-15 03:40:37,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38409.34 MB 2025-02-15 03:40:37,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38409.34 MB 2025-02-15 03:40:37,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:40:37,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35434.38 MB 2025-02-15 03:40:37,228 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 03:40:37,229 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:40:37,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:40:37,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:40:37,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:40:37,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:40:37,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24700.12 MB 2025-02-15 03:40:37,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33117.86 MB 2025-02-15 03:40:37,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-15 03:40:37,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38409.34 MB 2025-02-15 03:40:37,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46776.98 MB 2025-02-15 03:40:37,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 03:40:37,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33117.86 MB 2025-02-15 03:40:37,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 03:40:37,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:37,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:40:37,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:37,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:40:37,403 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:40:37,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:40:37,404 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:40:37,404 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:41:24,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:24,398 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:41:24,403 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:41:24,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:24,407 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1361, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:41:24,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:24,408 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1361, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:41:45,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:41:45,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:41:45,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.01 seconds 2025-02-15 03:41:45,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:45,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22452.37 MB 2025-02-15 03:41:45,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27269.53 MB 2025-02-15 03:41:45,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4817.16 MB 2025-02-15 03:41:45,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55144.61 MB 2025-02-15 03:41:45,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38017.17 MB 2025-02-15 03:41:45,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17127.44 MB 2025-02-15 03:41:45,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36227.10 MB 2025-02-15 03:41:45,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:41:45,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:41:45,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 03:41:45,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:45,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27269.53 MB 2025-02-15 03:41:45,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.27 MB 2025-02-15 03:41:45,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4416.27 MB 2025-02-15 03:41:45,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38017.17 MB 2025-02-15 03:41:45,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47578.09 MB 2025-02-15 03:41:45,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9560.92 MB 2025-02-15 03:41:45,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41646.36 MB 2025-02-15 03:41:47,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:41:47,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:41:47,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:41:47,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22853.27 MB 2025-02-15 03:41:47,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23384.11 MB 2025-02-15 03:41:47,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:41:47,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47578.09 MB 2025-02-15 03:41:47,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29016.20 MB 2025-02-15 03:41:47,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18561.89 MB 2025-02-15 03:41:47,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27362.65 MB 2025-02-15 03:41:47,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:41:47,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:41:47,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:41:47,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23384.11 MB 2025-02-15 03:41:47,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25273.64 MB 2025-02-15 03:41:47,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:41:47,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29016.20 MB 2025-02-15 03:41:47,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29959.91 MB 2025-02-15 03:41:47,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:41:47,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26691.07 MB 2025-02-15 03:41:47,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:41:47,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:41:47,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:41:47,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25273.64 MB 2025-02-15 03:41:47,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27515.50 MB 2025-02-15 03:41:47,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:41:47,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29959.91 MB 2025-02-15 03:41:47,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35622.22 MB 2025-02-15 03:41:47,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:41:47,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33059.78 MB 2025-02-15 03:41:47,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:41:47,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:41:47,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:41:47,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23384.11 MB 2025-02-15 03:41:47,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27515.50 MB 2025-02-15 03:41:47,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:41:47,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29016.20 MB 2025-02-15 03:41:47,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35622.22 MB 2025-02-15 03:41:47,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:41:47,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33059.78 MB 2025-02-15 03:41:47,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:41:47,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:41:47,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:41:47,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29049.04 MB 2025-02-15 03:41:47,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29816.04 MB 2025-02-15 03:41:47,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:41:47,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35622.22 MB 2025-02-15 03:41:47,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36039.56 MB 2025-02-15 03:41:47,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:41:47,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30523.83 MB 2025-02-15 03:41:47,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:41:47,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:41:47,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:41:47,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30228.93 MB 2025-02-15 03:41:47,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30457.82 MB 2025-02-15 03:41:47,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-15 03:41:47,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36039.56 MB 2025-02-15 03:41:47,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36039.56 MB 2025-02-15 03:41:47,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:41:47,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30690.65 MB 2025-02-15 03:41:47,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:41:47,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:41:47,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.43 seconds 2025-02-15 03:41:47,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:47,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17710.54 MB 2025-02-15 03:41:47,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30658.67 MB 2025-02-15 03:41:47,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12948.13 MB 2025-02-15 03:41:47,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55144.61 MB 2025-02-15 03:41:47,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36039.56 MB 2025-02-15 03:41:47,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19105.05 MB 2025-02-15 03:41:47,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30690.65 MB 2025-02-15 03:41:48,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:41:48,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:41:48,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:41:48,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:48,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30658.67 MB 2025-02-15 03:41:48,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22710.79 MB 2025-02-15 03:41:48,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7947.88 MB 2025-02-15 03:41:48,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36039.56 MB 2025-02-15 03:41:48,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36039.56 MB 2025-02-15 03:41:48,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:41:48,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33166.96 MB 2025-02-15 03:41:48,126 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 03:41:48,126 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:41:48,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:41:48,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:41:48,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:41:48,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:41:48,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22710.79 MB 2025-02-15 03:41:48,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31138.12 MB 2025-02-15 03:41:48,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 03:41:48,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36039.56 MB 2025-02-15 03:41:48,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44419.78 MB 2025-02-15 03:41:48,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 03:41:48,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31138.12 MB 2025-02-15 03:41:48,291 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 03:41:48,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:48,292 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:41:48,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:48,293 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:41:48,298 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:41:48,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:41:48,299 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:41:48,299 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:42:41,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:42:41,505 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:42:41,510 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:42:41,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:42:41,514 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1081, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:42:41,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:42:41,515 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1081, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:42:58,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:42:58,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:42:58,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.66 seconds 2025-02-15 03:42:58,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:42:58,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20501.29 MB 2025-02-15 03:42:58,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24326.89 MB 2025-02-15 03:42:58,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3825.60 MB 2025-02-15 03:42:58,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52800.00 MB 2025-02-15 03:42:58,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28659.68 MB 2025-02-15 03:42:58,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24140.32 MB 2025-02-15 03:42:58,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33143.55 MB 2025-02-15 03:42:58,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:42:58,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:42:58,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 03:42:58,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:42:58,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24326.89 MB 2025-02-15 03:42:58,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21398.68 MB 2025-02-15 03:42:58,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2928.20 MB 2025-02-15 03:42:58,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28659.68 MB 2025-02-15 03:42:58,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37901.83 MB 2025-02-15 03:42:58,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9242.15 MB 2025-02-15 03:42:58,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35667.63 MB 2025-02-15 03:43:00,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:43:00,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:43:00,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:43:00,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21398.68 MB 2025-02-15 03:43:00,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21929.52 MB 2025-02-15 03:43:00,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:43:00,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37901.83 MB 2025-02-15 03:43:00,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26956.79 MB 2025-02-15 03:43:00,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10945.04 MB 2025-02-15 03:43:00,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.07 MB 2025-02-15 03:43:00,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:43:00,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:43:00,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:43:00,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21929.52 MB 2025-02-15 03:43:00,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.06 MB 2025-02-15 03:43:00,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:43:00,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26956.79 MB 2025-02-15 03:43:00,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27900.51 MB 2025-02-15 03:43:00,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:43:00,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25236.49 MB 2025-02-15 03:43:00,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:43:00,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:43:00,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:43:00,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23819.06 MB 2025-02-15 03:43:00,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26060.91 MB 2025-02-15 03:43:00,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:43:00,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-15 03:43:00,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-15 03:43:00,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:43:00,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31605.19 MB 2025-02-15 03:43:00,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:43:00,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:43:00,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:43:00,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21929.52 MB 2025-02-15 03:43:00,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26060.91 MB 2025-02-15 03:43:00,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:43:00,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26956.79 MB 2025-02-15 03:43:00,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-15 03:43:00,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:43:00,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31605.19 MB 2025-02-15 03:43:00,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:43:00,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:43:00,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:43:00,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27594.46 MB 2025-02-15 03:43:00,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28361.46 MB 2025-02-15 03:43:00,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:43:00,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33562.82 MB 2025-02-15 03:43:00,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 03:43:00,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:43:00,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29069.25 MB 2025-02-15 03:43:00,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:43:00,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:43:00,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:43:00,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28774.35 MB 2025-02-15 03:43:00,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29004.06 MB 2025-02-15 03:43:00,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.71 MB 2025-02-15 03:43:00,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 03:43:00,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 03:43:00,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:43:00,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29205.20 MB 2025-02-15 03:43:00,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:43:00,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:43:00,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.09 seconds 2025-02-15 03:43:00,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16735.00 MB 2025-02-15 03:43:00,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29205.13 MB 2025-02-15 03:43:00,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12470.13 MB 2025-02-15 03:43:00,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52800.00 MB 2025-02-15 03:43:00,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 03:43:00,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18819.84 MB 2025-02-15 03:43:00,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29205.20 MB 2025-02-15 03:43:00,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:43:00,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:43:00,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:43:00,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29205.13 MB 2025-02-15 03:43:00,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21739.39 MB 2025-02-15 03:43:00,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7465.74 MB 2025-02-15 03:43:00,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 03:43:00,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 03:43:00,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:43:00,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31717.39 MB 2025-02-15 03:43:00,893 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:43:00,893 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:43:00,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:43:00,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:43:00,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:43:00,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:00,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21739.39 MB 2025-02-15 03:43:00,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30178.41 MB 2025-02-15 03:43:00,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:43:00,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 03:43:00,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42370.86 MB 2025-02-15 03:43:00,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:43:00,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30178.41 MB 2025-02-15 03:43:01,061 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:43:01,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:01,062 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:43:01,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:01,063 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:43:01,068 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:43:01,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:01,069 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:43:01,069 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:43:15,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:15,076 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:43:15,081 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:43:15,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:15,084 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1250, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:43:15,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:15,085 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1250, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:43:34,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:43:34,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:43:34,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.42 seconds 2025-02-15 03:43:34,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:34,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21678.91 MB 2025-02-15 03:43:34,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.59 MB 2025-02-15 03:43:34,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4423.68 MB 2025-02-15 03:43:34,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54955.87 MB 2025-02-15 03:43:34,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37648.07 MB 2025-02-15 03:43:34,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17307.80 MB 2025-02-15 03:43:34,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35000.65 MB 2025-02-15 03:43:34,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:43:34,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:43:34,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:43:34,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:34,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.59 MB 2025-02-15 03:43:34,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22276.21 MB 2025-02-15 03:43:34,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3826.38 MB 2025-02-15 03:43:34,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37648.07 MB 2025-02-15 03:43:34,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46372.23 MB 2025-02-15 03:43:34,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8724.15 MB 2025-02-15 03:43:34,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39220.98 MB 2025-02-15 03:43:36,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:43:36,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:43:36,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:43:36,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22276.21 MB 2025-02-15 03:43:36,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22807.05 MB 2025-02-15 03:43:36,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:43:36,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46372.23 MB 2025-02-15 03:43:36,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 03:43:36,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17332.96 MB 2025-02-15 03:43:36,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26785.60 MB 2025-02-15 03:43:36,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:43:36,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:43:36,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:43:36,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-15 03:43:36,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24696.59 MB 2025-02-15 03:43:36,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:43:36,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 03:43:36,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 03:43:36,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:43:36,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26114.02 MB 2025-02-15 03:43:36,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:43:36,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:43:36,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:43:36,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24696.59 MB 2025-02-15 03:43:36,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-15 03:43:36,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:43:36,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 03:43:36,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:43:36,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:43:36,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-15 03:43:36,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:43:36,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:43:36,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:43:36,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-15 03:43:36,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-15 03:43:36,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:43:36,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 03:43:36,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:43:36,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:43:36,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-15 03:43:36,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:43:36,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:43:36,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:43:36,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28471.98 MB 2025-02-15 03:43:36,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29238.99 MB 2025-02-15 03:43:36,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:43:36,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-15 03:43:36,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-15 03:43:36,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:43:36,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29946.78 MB 2025-02-15 03:43:36,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:43:36,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:43:36,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:43:36,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29651.88 MB 2025-02-15 03:43:36,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29880.27 MB 2025-02-15 03:43:36,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-15 03:43:36,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35116.81 MB 2025-02-15 03:43:36,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-15 03:43:36,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:43:36,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30112.28 MB 2025-02-15 03:43:36,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:43:36,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:43:36,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.84 seconds 2025-02-15 03:43:36,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:36,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17323.81 MB 2025-02-15 03:43:36,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30081.12 MB 2025-02-15 03:43:36,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12757.32 MB 2025-02-15 03:43:36,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54955.87 MB 2025-02-15 03:43:36,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-15 03:43:36,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19839.06 MB 2025-02-15 03:43:36,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30112.28 MB 2025-02-15 03:43:37,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:43:37,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:43:37,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:43:37,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:37,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30081.12 MB 2025-02-15 03:43:37,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22316.93 MB 2025-02-15 03:43:37,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7764.19 MB 2025-02-15 03:43:37,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35116.81 MB 2025-02-15 03:43:37,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-15 03:43:37,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:43:37,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32583.27 MB 2025-02-15 03:43:37,210 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 03:43:37,211 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:43:37,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:43:37,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:43:37,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:43:37,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:43:37,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22316.93 MB 2025-02-15 03:43:37,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30724.66 MB 2025-02-15 03:43:37,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 03:43:37,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35116.81 MB 2025-02-15 03:43:37,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39296.43 MB 2025-02-15 03:43:37,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 03:43:37,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30724.66 MB 2025-02-15 03:43:37,379 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 03:43:37,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:37,381 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:43:37,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:37,382 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:43:37,387 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:43:37,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:43:37,389 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:43:37,389 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:45:24,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:24,379 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:45:24,384 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:45:24,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:24,388 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 298, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:45:24,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:24,389 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 298, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:45:28,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:45:28,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:45:28,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.58 seconds 2025-02-15 03:45:28,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:28,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15045.22 MB 2025-02-15 03:45:28,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16099.82 MB 2025-02-15 03:45:28,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1054.61 MB 2025-02-15 03:45:28,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 03:45:28,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-15 03:45:28,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26705.13 MB 2025-02-15 03:45:28,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24969.57 MB 2025-02-15 03:45:28,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:45:28,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:45:28,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:45:28,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:28,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16099.82 MB 2025-02-15 03:45:28,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16527.58 MB 2025-02-15 03:45:28,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 427.75 MB 2025-02-15 03:45:28,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-15 03:45:28,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22961.72 MB 2025-02-15 03:45:28,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2011.17 MB 2025-02-15 03:45:28,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20118.13 MB 2025-02-15 03:45:30,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:45:30,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:45:30,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.36 seconds 2025-02-15 03:45:30,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16527.58 MB 2025-02-15 03:45:30,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16907.13 MB 2025-02-15 03:45:30,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.55 MB 2025-02-15 03:45:30,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22961.72 MB 2025-02-15 03:45:30,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20380.12 MB 2025-02-15 03:45:30,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2581.59 MB 2025-02-15 03:45:30,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20868.13 MB 2025-02-15 03:45:30,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:45:30,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:45:30,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:45:30,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16907.13 MB 2025-02-15 03:45:30,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18258.61 MB 2025-02-15 03:45:30,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1351.48 MB 2025-02-15 03:45:30,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20380.12 MB 2025-02-15 03:45:30,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21055.41 MB 2025-02-15 03:45:30,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 675.28 MB 2025-02-15 03:45:30,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19272.08 MB 2025-02-15 03:45:30,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:45:30,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:45:30,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:45:30,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18258.61 MB 2025-02-15 03:45:30,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19861.56 MB 2025-02-15 03:45:30,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1602.95 MB 2025-02-15 03:45:30,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21055.41 MB 2025-02-15 03:45:30,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25107.10 MB 2025-02-15 03:45:30,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4051.70 MB 2025-02-15 03:45:30,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23828.32 MB 2025-02-15 03:45:30,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:45:30,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:45:30,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 03:45:30,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16907.13 MB 2025-02-15 03:45:30,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19861.56 MB 2025-02-15 03:45:30,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2954.43 MB 2025-02-15 03:45:30,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20380.12 MB 2025-02-15 03:45:30,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25107.10 MB 2025-02-15 03:45:30,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4726.98 MB 2025-02-15 03:45:30,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23828.32 MB 2025-02-15 03:45:30,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:45:30,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:45:30,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:45:30,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20958.04 MB 2025-02-15 03:45:30,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21506.71 MB 2025-02-15 03:45:30,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 548.67 MB 2025-02-15 03:45:30,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25107.10 MB 2025-02-15 03:45:30,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-15 03:45:30,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 293.60 MB 2025-02-15 03:45:30,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22012.78 MB 2025-02-15 03:45:30,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:45:30,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:45:30,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:45:30,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21801.93 MB 2025-02-15 03:45:30,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22021.28 MB 2025-02-15 03:45:30,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.35 MB 2025-02-15 03:45:30,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-15 03:45:30,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-15 03:45:30,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:45:30,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22135.92 MB 2025-02-15 03:45:30,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:45:30,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:45:30,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.27 seconds 2025-02-15 03:45:30,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14006.96 MB 2025-02-15 03:45:30,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22222.13 MB 2025-02-15 03:45:30,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8215.17 MB 2025-02-15 03:45:30,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 03:45:30,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-15 03:45:30,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22254.98 MB 2025-02-15 03:45:30,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22222.13 MB 2025-02-15 03:45:30,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:45:30,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:45:30,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:45:30,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22222.13 MB 2025-02-15 03:45:30,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25232.11 MB 2025-02-15 03:45:30,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.98 MB 2025-02-15 03:45:30,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-15 03:45:30,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26877.10 MB 2025-02-15 03:45:30,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-15 03:45:30,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25533.51 MB 2025-02-15 03:45:30,942 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 03:45:30,942 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-15 03:45:30,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:45:30,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:45:30,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:45:30,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:45:30,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18469.42 MB 2025-02-15 03:45:30,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26896.76 MB 2025-02-15 03:45:30,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 03:45:30,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26877.10 MB 2025-02-15 03:45:30,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35257.32 MB 2025-02-15 03:45:30,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 03:45:30,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26896.76 MB 2025-02-15 03:45:31,109 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 03:45:31,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:31,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:45:31,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:31,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:45:31,116 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:45:31,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:31,117 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:45:31,117 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-15 03:45:41,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:41,235 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:45:41,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:45:41,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:41,243 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2493, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:45:41,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:45:41,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2493, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:46:19,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:46:19,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:46:19,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.57 seconds 2025-02-15 03:46:19,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:19,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30340.33 MB 2025-02-15 03:46:19,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39163.05 MB 2025-02-15 03:46:19,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8822.72 MB 2025-02-15 03:46:19,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61014.54 MB 2025-02-15 03:46:19,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42679.14 MB 2025-02-15 03:46:19,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18335.40 MB 2025-02-15 03:46:19,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47985.64 MB 2025-02-15 03:46:20,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:46:20,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:46:20,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.37 seconds 2025-02-15 03:46:20,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:20,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39163.05 MB 2025-02-15 03:46:20,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28739.23 MB 2025-02-15 03:46:20,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10423.82 MB 2025-02-15 03:46:20,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42679.14 MB 2025-02-15 03:46:20,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61085.84 MB 2025-02-15 03:46:20,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18406.70 MB 2025-02-15 03:46:20,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63652.03 MB 2025-02-15 03:46:22,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:46:22,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:46:22,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 03:46:22,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28739.23 MB 2025-02-15 03:46:22,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29270.07 MB 2025-02-15 03:46:22,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:46:22,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61085.84 MB 2025-02-15 03:46:22,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31568.43 MB 2025-02-15 03:46:22,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29517.41 MB 2025-02-15 03:46:22,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33248.62 MB 2025-02-15 03:46:22,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:46:22,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:46:22,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:46:22,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29270.07 MB 2025-02-15 03:46:22,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31159.60 MB 2025-02-15 03:46:22,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:46:22,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31568.43 MB 2025-02-15 03:46:22,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34399.58 MB 2025-02-15 03:46:22,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:46:22,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32577.03 MB 2025-02-15 03:46:22,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:46:22,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:46:22,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:46:22,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31159.60 MB 2025-02-15 03:46:22,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33401.46 MB 2025-02-15 03:46:22,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:46:22,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34399.58 MB 2025-02-15 03:46:22,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40533.75 MB 2025-02-15 03:46:22,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:46:22,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38945.74 MB 2025-02-15 03:46:22,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:46:22,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:46:22,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 03:46:22,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29270.07 MB 2025-02-15 03:46:22,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33401.46 MB 2025-02-15 03:46:22,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:46:22,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31568.43 MB 2025-02-15 03:46:22,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40533.75 MB 2025-02-15 03:46:22,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 03:46:22,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38945.74 MB 2025-02-15 03:46:22,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:46:22,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:46:22,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:46:22,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34935.00 MB 2025-02-15 03:46:22,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35702.00 MB 2025-02-15 03:46:22,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:46:22,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40533.75 MB 2025-02-15 03:46:22,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40948.99 MB 2025-02-15 03:46:22,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:46:22,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36409.79 MB 2025-02-15 03:46:22,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:46:22,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:46:22,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:46:22,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36114.89 MB 2025-02-15 03:46:22,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36342.50 MB 2025-02-15 03:46:22,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.60 MB 2025-02-15 03:46:22,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40948.99 MB 2025-02-15 03:46:22,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40948.99 MB 2025-02-15 03:46:22,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:46:22,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36570.28 MB 2025-02-15 03:46:22,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:46:22,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:46:22,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.34 seconds 2025-02-15 03:46:22,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21654.52 MB 2025-02-15 03:46:22,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36543.03 MB 2025-02-15 03:46:22,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14888.51 MB 2025-02-15 03:46:22,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52326.04 MB 2025-02-15 03:46:22,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40948.99 MB 2025-02-15 03:46:22,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11377.05 MB 2025-02-15 03:46:22,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36570.28 MB 2025-02-15 03:46:22,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:46:22,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:46:22,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:46:22,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36543.03 MB 2025-02-15 03:46:22,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26650.32 MB 2025-02-15 03:46:22,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9892.70 MB 2025-02-15 03:46:22,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40948.99 MB 2025-02-15 03:46:22,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40948.99 MB 2025-02-15 03:46:22,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:46:22,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39047.94 MB 2025-02-15 03:46:22,878 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 03:46:22,879 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:46:22,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:46:22,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:46:22,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:46:22,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:46:22,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26650.32 MB 2025-02-15 03:46:22,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35066.93 MB 2025-02-15 03:46:22,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 03:46:22,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40948.99 MB 2025-02-15 03:46:22,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45132.81 MB 2025-02-15 03:46:22,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 03:46:22,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35066.93 MB 2025-02-15 03:46:23,045 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 03:46:23,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:46:23,046 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:46:23,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:46:23,047 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:46:23,052 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:46:23,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:46:23,053 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:46:23,053 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:48:38,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:38,523 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:48:38,529 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:48:38,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:38,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:48:38,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:38,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:48:41,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:48:41,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:48:41,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-15 03:48:41,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:41,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-15 03:48:41,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-15 03:48:41,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-15 03:48:41,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53500.44 MB 2025-02-15 03:48:41,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19144.90 MB 2025-02-15 03:48:41,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34355.54 MB 2025-02-15 03:48:41,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-15 03:48:41,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:48:41,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:48:41,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:48:41,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:41,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-15 03:48:41,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15178.74 MB 2025-02-15 03:48:41,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.71 MB 2025-02-15 03:48:41,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19144.90 MB 2025-02-15 03:48:41,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19144.90 MB 2025-02-15 03:48:41,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:48:41,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17409.24 MB 2025-02-15 03:48:42,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:48:42,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:48:42,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 03:48:42,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15178.74 MB 2025-02-15 03:48:42,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15413.64 MB 2025-02-15 03:48:42,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 03:48:42,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19144.90 MB 2025-02-15 03:48:42,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17968.40 MB 2025-02-15 03:48:42,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1176.50 MB 2025-02-15 03:48:42,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19349.43 MB 2025-02-15 03:48:42,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:48:42,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:48:42,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:48:42,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15413.57 MB 2025-02-15 03:48:42,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16249.49 MB 2025-02-15 03:48:42,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 03:48:42,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17968.40 MB 2025-02-15 03:48:42,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18807.26 MB 2025-02-15 03:48:42,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 838.86 MB 2025-02-15 03:48:42,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16876.70 MB 2025-02-15 03:48:42,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:48:42,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:48:42,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:48:42,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16249.49 MB 2025-02-15 03:48:42,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17241.54 MB 2025-02-15 03:48:42,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 03:48:42,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18807.26 MB 2025-02-15 03:48:42,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-15 03:48:42,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2306.87 MB 2025-02-15 03:48:42,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19695.77 MB 2025-02-15 03:48:42,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:48:42,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:48:42,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:48:42,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15413.57 MB 2025-02-15 03:48:42,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17241.54 MB 2025-02-15 03:48:42,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 03:48:42,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17968.40 MB 2025-02-15 03:48:42,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-15 03:48:42,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3145.73 MB 2025-02-15 03:48:42,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19695.77 MB 2025-02-15 03:48:42,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:48:42,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:48:42,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:48:42,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17920.14 MB 2025-02-15 03:48:42,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18260.45 MB 2025-02-15 03:48:42,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 03:48:42,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21114.13 MB 2025-02-15 03:48:42,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21294.48 MB 2025-02-15 03:48:42,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 03:48:42,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18579.92 MB 2025-02-15 03:48:42,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:48:42,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:48:42,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:48:42,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18443.16 MB 2025-02-15 03:48:42,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18667.06 MB 2025-02-15 03:48:42,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.89 MB 2025-02-15 03:48:42,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21294.48 MB 2025-02-15 03:48:42,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21294.48 MB 2025-02-15 03:48:42,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:48:42,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18699.52 MB 2025-02-15 03:48:42,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:48:42,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:48:42,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.91 seconds 2025-02-15 03:48:42,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-15 03:48:42,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18868.13 MB 2025-02-15 03:48:42,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5251.38 MB 2025-02-15 03:48:42,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53500.44 MB 2025-02-15 03:48:42,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21294.48 MB 2025-02-15 03:48:42,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32205.96 MB 2025-02-15 03:48:42,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18868.13 MB 2025-02-15 03:48:42,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:48:42,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:48:42,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:48:42,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18868.13 MB 2025-02-15 03:48:42,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17569.39 MB 2025-02-15 03:48:42,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1298.74 MB 2025-02-15 03:48:42,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21294.48 MB 2025-02-15 03:48:42,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21294.48 MB 2025-02-15 03:48:42,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:48:42,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19102.55 MB 2025-02-15 03:48:42,726 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:48:42,727 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:48:42,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:48:42,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:48:42,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:48:42,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:48:42,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17569.39 MB 2025-02-15 03:48:42,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.41 MB 2025-02-15 03:48:42,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:48:42,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21294.48 MB 2025-02-15 03:48:42,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31784.44 MB 2025-02-15 03:48:42,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 03:48:42,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26008.41 MB 2025-02-15 03:48:42,894 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:48:42,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:42,895 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:48:42,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:42,896 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:48:42,901 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:48:42,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:48:42,902 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:48:42,902 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:49:11,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:11,982 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:49:11,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:49:11,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:11,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2715, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:49:11,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:11,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2715, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:49:53,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:49:53,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:49:53,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.81 seconds 2025-02-15 03:49:53,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:53,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31889.21 MB 2025-02-15 03:49:53,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41498.36 MB 2025-02-15 03:49:53,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9609.15 MB 2025-02-15 03:49:53,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63289.95 MB 2025-02-15 03:49:53,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45006.98 MB 2025-02-15 03:49:53,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18282.97 MB 2025-02-15 03:49:53,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51106.59 MB 2025-02-15 03:49:54,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:49:54,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:49:54,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 03:49:54,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:54,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41498.36 MB 2025-02-15 03:49:54,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29894.83 MB 2025-02-15 03:49:54,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11603.53 MB 2025-02-15 03:49:54,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45006.98 MB 2025-02-15 03:49:54,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64124.62 MB 2025-02-15 03:49:54,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19117.64 MB 2025-02-15 03:49:54,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67057.91 MB 2025-02-15 03:49:55,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:49:55,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:49:55,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:49:55,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:55,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29894.83 MB 2025-02-15 03:49:55,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30425.67 MB 2025-02-15 03:49:55,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:49:55,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64124.62 MB 2025-02-15 03:49:55,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32717.67 MB 2025-02-15 03:49:55,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31406.95 MB 2025-02-15 03:49:55,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34404.22 MB 2025-02-15 03:49:56,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:49:56,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:49:56,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:49:56,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30425.67 MB 2025-02-15 03:49:56,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32315.20 MB 2025-02-15 03:49:56,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:49:56,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32717.67 MB 2025-02-15 03:49:56,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-15 03:49:56,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 03:49:56,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33732.63 MB 2025-02-15 03:49:56,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:49:56,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:49:56,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:49:56,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32315.20 MB 2025-02-15 03:49:56,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34557.06 MB 2025-02-15 03:49:56,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:49:56,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-15 03:49:56,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41682.99 MB 2025-02-15 03:49:56,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:49:56,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40101.34 MB 2025-02-15 03:49:56,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:49:56,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:49:56,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:49:56,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30425.67 MB 2025-02-15 03:49:56,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34557.06 MB 2025-02-15 03:49:56,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:49:56,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32717.67 MB 2025-02-15 03:49:56,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41682.99 MB 2025-02-15 03:49:56,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 03:49:56,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40101.34 MB 2025-02-15 03:49:56,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:49:56,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:49:56,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:49:56,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36090.60 MB 2025-02-15 03:49:56,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36857.60 MB 2025-02-15 03:49:56,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:49:56,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41682.99 MB 2025-02-15 03:49:56,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42100.33 MB 2025-02-15 03:49:56,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:49:56,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37565.39 MB 2025-02-15 03:49:56,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:49:56,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:49:56,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:49:56,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37270.49 MB 2025-02-15 03:49:56,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37498.86 MB 2025-02-15 03:49:56,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 03:49:56,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42100.33 MB 2025-02-15 03:49:56,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42100.33 MB 2025-02-15 03:49:56,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:49:56,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37728.94 MB 2025-02-15 03:49:56,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:49:56,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:49:56,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.40 seconds 2025-02-15 03:49:56,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22428.96 MB 2025-02-15 03:49:56,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37699.15 MB 2025-02-15 03:49:56,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15270.19 MB 2025-02-15 03:49:56,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53829.70 MB 2025-02-15 03:49:56,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42100.33 MB 2025-02-15 03:49:56,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11729.37 MB 2025-02-15 03:49:56,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37728.94 MB 2025-02-15 03:49:56,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:49:56,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:49:56,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:49:56,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37699.15 MB 2025-02-15 03:49:56,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27421.72 MB 2025-02-15 03:49:56,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10277.43 MB 2025-02-15 03:49:56,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42100.33 MB 2025-02-15 03:49:56,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42100.33 MB 2025-02-15 03:49:56,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:49:56,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40200.99 MB 2025-02-15 03:49:56,686 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 03:49:56,686 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:49:56,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:49:56,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:49:56,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:49:56,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:49:56,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27421.72 MB 2025-02-15 03:49:56,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35827.38 MB 2025-02-15 03:49:56,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 03:49:56,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42100.33 MB 2025-02-15 03:49:56,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46279.95 MB 2025-02-15 03:49:56,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 03:49:56,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35827.38 MB 2025-02-15 03:49:56,851 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 03:49:56,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:56,853 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:49:56,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:56,854 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:49:56,858 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:49:56,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:49:56,859 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:49:56,859 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:50:07,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:07,484 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:50:07,492 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:50:07,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:07,500 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 733, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:50:07,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:07,502 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 733, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:50:19,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:50:19,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:50:19,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.54 seconds 2025-02-15 03:50:19,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:19,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18076.37 MB 2025-02-15 03:50:19,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20670.54 MB 2025-02-15 03:50:19,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2594.18 MB 2025-02-15 03:50:19,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54639.20 MB 2025-02-15 03:50:19,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24668.80 MB 2025-02-15 03:50:19,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29970.40 MB 2025-02-15 03:50:19,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29586.98 MB 2025-02-15 03:50:19,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:50:19,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:50:19,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 03:50:19,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:19,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20670.54 MB 2025-02-15 03:50:19,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19589.54 MB 2025-02-15 03:50:19,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1081.01 MB 2025-02-15 03:50:19,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24668.80 MB 2025-02-15 03:50:19,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31492.93 MB 2025-02-15 03:50:19,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6824.13 MB 2025-02-15 03:50:19,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29594.98 MB 2025-02-15 03:50:21,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:50:21,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:50:21,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:50:21,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19589.54 MB 2025-02-15 03:50:21,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20120.38 MB 2025-02-15 03:50:21,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:50:21,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31492.93 MB 2025-02-15 03:50:21,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24199.04 MB 2025-02-15 03:50:21,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7293.89 MB 2025-02-15 03:50:21,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24099.96 MB 2025-02-15 03:50:21,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:50:21,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:50:21,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:50:21,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-15 03:50:21,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22009.91 MB 2025-02-15 03:50:21,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:50:21,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24199.04 MB 2025-02-15 03:50:21,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26086.47 MB 2025-02-15 03:50:21,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 03:50:21,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23427.34 MB 2025-02-15 03:50:21,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:50:21,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:50:21,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:50:21,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22009.91 MB 2025-02-15 03:50:21,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-15 03:50:21,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:50:21,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26086.47 MB 2025-02-15 03:50:21,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-15 03:50:21,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 03:50:21,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.64 MB 2025-02-15 03:50:21,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:50:21,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:50:21,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:50:21,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-15 03:50:21,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-15 03:50:21,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:50:21,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24199.04 MB 2025-02-15 03:50:21,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-15 03:50:21,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 03:50:21,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.64 MB 2025-02-15 03:50:21,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:50:21,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:50:21,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:50:21,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25785.90 MB 2025-02-15 03:50:21,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26552.90 MB 2025-02-15 03:50:21,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:50:21,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31748.78 MB 2025-02-15 03:50:21,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32164.02 MB 2025-02-15 03:50:21,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:50:21,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27260.69 MB 2025-02-15 03:50:21,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:50:21,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:50:21,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:50:21,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26965.79 MB 2025-02-15 03:50:21,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27192.75 MB 2025-02-15 03:50:21,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.96 MB 2025-02-15 03:50:21,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32164.02 MB 2025-02-15 03:50:21,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32164.02 MB 2025-02-15 03:50:21,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:50:21,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27378.04 MB 2025-02-15 03:50:21,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:50:21,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:50:21,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.95 seconds 2025-02-15 03:50:21,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15522.54 MB 2025-02-15 03:50:21,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27393.82 MB 2025-02-15 03:50:21,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11871.29 MB 2025-02-15 03:50:21,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54639.20 MB 2025-02-15 03:50:21,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32164.02 MB 2025-02-15 03:50:21,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22475.18 MB 2025-02-15 03:50:21,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27393.82 MB 2025-02-15 03:50:21,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:50:21,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:50:21,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:50:21,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27393.82 MB 2025-02-15 03:50:21,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20527.52 MB 2025-02-15 03:50:21,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6866.31 MB 2025-02-15 03:50:21,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32164.02 MB 2025-02-15 03:50:21,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32164.02 MB 2025-02-15 03:50:21,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:50:21,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29905.49 MB 2025-02-15 03:50:21,750 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:50:21,751 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:50:21,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:50:21,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:50:21,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:50:21,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:50:21,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20527.52 MB 2025-02-15 03:50:21,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.54 MB 2025-02-15 03:50:21,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:50:21,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32164.02 MB 2025-02-15 03:50:21,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40554.73 MB 2025-02-15 03:50:21,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:50:21,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28966.54 MB 2025-02-15 03:50:21,915 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:50:21,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:21,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:50:21,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:21,917 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:50:21,922 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:50:21,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:50:21,923 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:50:21,923 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:51:16,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:16,895 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:51:16,900 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:51:16,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:16,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 155, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:51:16,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:16,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 155, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:51:19,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:51:19,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:51:19,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.41 seconds 2025-02-15 03:51:19,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:19,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14048.77 MB 2025-02-15 03:51:19,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14597.31 MB 2025-02-15 03:51:19,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 548.54 MB 2025-02-15 03:51:19,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53139.73 MB 2025-02-15 03:51:19,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18129.88 MB 2025-02-15 03:51:19,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35009.86 MB 2025-02-15 03:51:19,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23520.14 MB 2025-02-15 03:51:19,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:51:19,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:51:19,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:51:19,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:19,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14597.31 MB 2025-02-15 03:51:19,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14820.93 MB 2025-02-15 03:51:19,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.63 MB 2025-02-15 03:51:19,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18129.88 MB 2025-02-15 03:51:19,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18658.36 MB 2025-02-15 03:51:19,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 528.48 MB 2025-02-15 03:51:19,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16690.23 MB 2025-02-15 03:51:20,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:51:20,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:51:20,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 03:51:20,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14820.93 MB 2025-02-15 03:51:20,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15018.67 MB 2025-02-15 03:51:20,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.74 MB 2025-02-15 03:51:20,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18658.36 MB 2025-02-15 03:51:20,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18658.36 MB 2025-02-15 03:51:20,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:51:20,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18991.62 MB 2025-02-15 03:51:20,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:51:20,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:51:20,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 03:51:20,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-15 03:51:20,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15722.29 MB 2025-02-15 03:51:20,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 703.68 MB 2025-02-15 03:51:20,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18658.36 MB 2025-02-15 03:51:20,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18658.36 MB 2025-02-15 03:51:20,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:51:20,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16250.29 MB 2025-02-15 03:51:20,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:51:20,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:51:20,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 03:51:20,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15722.29 MB 2025-02-15 03:51:20,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.42 MB 2025-02-15 03:51:20,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.13 MB 2025-02-15 03:51:20,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18658.36 MB 2025-02-15 03:51:20,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19715.33 MB 2025-02-15 03:51:20,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1056.96 MB 2025-02-15 03:51:20,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18624.98 MB 2025-02-15 03:51:20,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:51:20,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:51:20,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 03:51:20,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-15 03:51:20,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.42 MB 2025-02-15 03:51:20,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1538.81 MB 2025-02-15 03:51:20,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18658.36 MB 2025-02-15 03:51:20,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19715.33 MB 2025-02-15 03:51:20,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1056.96 MB 2025-02-15 03:51:20,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18624.98 MB 2025-02-15 03:51:20,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:51:20,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:51:20,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:51:20,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17128.66 MB 2025-02-15 03:51:20,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17414.37 MB 2025-02-15 03:51:20,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.71 MB 2025-02-15 03:51:20,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19715.33 MB 2025-02-15 03:51:20,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 03:51:20,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 03:51:20,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17688.15 MB 2025-02-15 03:51:20,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:51:20,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:51:20,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:51:20,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17568.18 MB 2025-02-15 03:51:20,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17784.90 MB 2025-02-15 03:51:20,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.72 MB 2025-02-15 03:51:20,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19868.42 MB 2025-02-15 03:51:20,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 03:51:20,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:51:20,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17800.28 MB 2025-02-15 03:51:20,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:51:20,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:51:20,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-15 03:51:20,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13508.74 MB 2025-02-15 03:51:20,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17985.65 MB 2025-02-15 03:51:20,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4476.91 MB 2025-02-15 03:51:20,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53139.73 MB 2025-02-15 03:51:20,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 03:51:20,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33271.32 MB 2025-02-15 03:51:20,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17985.65 MB 2025-02-15 03:51:20,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:51:20,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:51:20,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 03:51:20,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17985.65 MB 2025-02-15 03:51:20,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17323.73 MB 2025-02-15 03:51:20,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -661.92 MB 2025-02-15 03:51:20,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19868.42 MB 2025-02-15 03:51:20,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 03:51:20,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:51:20,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18988.72 MB 2025-02-15 03:51:20,612 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 03:51:20,612 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:51:20,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:51:20,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:51:20,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:51:20,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:51:20,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17323.73 MB 2025-02-15 03:51:20,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25749.91 MB 2025-02-15 03:51:20,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 03:51:20,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19868.42 MB 2025-02-15 03:51:20,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30339.50 MB 2025-02-15 03:51:20,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 03:51:20,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25749.91 MB 2025-02-15 03:51:20,889 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 03:51:20,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:20,892 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:51:20,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:20,894 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:51:20,902 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:51:20,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:51:20,904 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:51:20,904 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:52:16,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:16,154 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:52:16,159 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:52:16,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:16,164 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1669, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:52:16,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:16,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1669, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:52:41,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:52:41,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:52:41,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.46 seconds 2025-02-15 03:52:41,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:41,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24598.57 MB 2025-02-15 03:52:41,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30505.06 MB 2025-02-15 03:52:41,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5906.50 MB 2025-02-15 03:52:41,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38715.52 MB 2025-02-15 03:52:41,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 03:52:41,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 421.53 MB 2025-02-15 03:52:41,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39505.75 MB 2025-02-15 03:52:41,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:52:41,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:52:41,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:52:41,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:41,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30505.06 MB 2025-02-15 03:52:41,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24454.46 MB 2025-02-15 03:52:41,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6050.60 MB 2025-02-15 03:52:41,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 03:52:41,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43100.67 MB 2025-02-15 03:52:41,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3963.62 MB 2025-02-15 03:52:41,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39786.43 MB 2025-02-15 03:52:43,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:52:43,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:52:43,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 03:52:43,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:43,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24454.46 MB 2025-02-15 03:52:43,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24985.30 MB 2025-02-15 03:52:43,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:52:43,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43100.67 MB 2025-02-15 03:52:43,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30456.94 MB 2025-02-15 03:52:43,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12643.73 MB 2025-02-15 03:52:43,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28963.85 MB 2025-02-15 03:52:43,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:52:43,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:52:43,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:52:43,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:43,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-15 03:52:43,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26874.84 MB 2025-02-15 03:52:43,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:52:43,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30456.94 MB 2025-02-15 03:52:43,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30456.94 MB 2025-02-15 03:52:43,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:52:43,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28292.27 MB 2025-02-15 03:52:43,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:52:43,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:52:43,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:52:43,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:43,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26874.84 MB 2025-02-15 03:52:43,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-15 03:52:43,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:52:43,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30456.94 MB 2025-02-15 03:52:43,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36591.11 MB 2025-02-15 03:52:43,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:52:43,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-15 03:52:43,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:52:43,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:52:43,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:52:43,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:43,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-15 03:52:43,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-15 03:52:43,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:52:43,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30456.94 MB 2025-02-15 03:52:43,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36591.11 MB 2025-02-15 03:52:43,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 03:52:43,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-15 03:52:44,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:52:44,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:52:44,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:52:44,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:44,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30650.24 MB 2025-02-15 03:52:44,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31417.24 MB 2025-02-15 03:52:44,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:52:44,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36591.11 MB 2025-02-15 03:52:44,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 03:52:44,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 03:52:44,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32125.03 MB 2025-02-15 03:52:44,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:52:44,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:52:44,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:52:44,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:44,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31830.13 MB 2025-02-15 03:52:44,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32060.16 MB 2025-02-15 03:52:44,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.03 MB 2025-02-15 03:52:44,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37008.44 MB 2025-02-15 03:52:44,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 03:52:44,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:52:44,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32251.72 MB 2025-02-15 03:52:44,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:52:44,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:52:44,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.86 seconds 2025-02-15 03:52:44,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:44,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18783.64 MB 2025-02-15 03:52:44,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32261.23 MB 2025-02-15 03:52:44,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13477.59 MB 2025-02-15 03:52:44,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38715.52 MB 2025-02-15 03:52:44,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 03:52:44,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1707.08 MB 2025-02-15 03:52:44,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32261.23 MB 2025-02-15 03:52:44,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:52:44,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:52:44,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:52:44,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:44,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32261.23 MB 2025-02-15 03:52:44,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23788.03 MB 2025-02-15 03:52:44,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8473.20 MB 2025-02-15 03:52:44,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37008.44 MB 2025-02-15 03:52:44,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 03:52:44,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:52:44,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34772.90 MB 2025-02-15 03:52:44,313 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:52:44,314 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:52:44,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:52:44,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:52:44,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:52:44,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:52:44,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23788.03 MB 2025-02-15 03:52:44,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32227.05 MB 2025-02-15 03:52:44,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:52:44,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37008.44 MB 2025-02-15 03:52:44,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45399.15 MB 2025-02-15 03:52:44,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:52:44,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.05 MB 2025-02-15 03:52:44,482 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:52:44,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:44,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:52:44,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:44,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:52:44,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:52:44,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:52:44,490 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:52:44,490 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:53:05,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:05,476 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:53:05,481 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:53:05,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:05,484 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1241, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:53:05,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:05,485 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1241, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:53:24,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:53:24,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:53:24,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.05 seconds 2025-02-15 03:53:24,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:24,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21616.19 MB 2025-02-15 03:53:24,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.02 MB 2025-02-15 03:53:24,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4391.83 MB 2025-02-15 03:53:24,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57984.16 MB 2025-02-15 03:53:24,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37631.30 MB 2025-02-15 03:53:24,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20352.86 MB 2025-02-15 03:53:24,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34937.94 MB 2025-02-15 03:53:24,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:53:24,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:53:24,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:53:24,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:24,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26008.02 MB 2025-02-15 03:53:24,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22229.42 MB 2025-02-15 03:53:24,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3778.60 MB 2025-02-15 03:53:24,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37631.30 MB 2025-02-15 03:53:24,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44109.40 MB 2025-02-15 03:53:24,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6478.10 MB 2025-02-15 03:53:24,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38895.13 MB 2025-02-15 03:53:26,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:53:26,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:53:26,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 03:53:26,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22229.42 MB 2025-02-15 03:53:26,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22760.26 MB 2025-02-15 03:53:26,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:53:26,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44109.40 MB 2025-02-15 03:53:26,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33237.76 MB 2025-02-15 03:53:26,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10871.64 MB 2025-02-15 03:53:26,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26738.81 MB 2025-02-15 03:53:26,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:53:26,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:53:26,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:53:26,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-15 03:53:26,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24649.80 MB 2025-02-15 03:53:26,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:53:26,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 03:53:26,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33237.76 MB 2025-02-15 03:53:26,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:53:26,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26067.23 MB 2025-02-15 03:53:26,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:53:26,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:53:26,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:53:26,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24649.80 MB 2025-02-15 03:53:26,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-15 03:53:26,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:53:26,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 03:53:26,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33709.62 MB 2025-02-15 03:53:26,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 03:53:26,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-15 03:53:26,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:53:26,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:53:26,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:53:26,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-15 03:53:26,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-15 03:53:26,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:53:26,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 03:53:26,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33709.62 MB 2025-02-15 03:53:26,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 03:53:26,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-15 03:53:26,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:53:26,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:53:26,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:53:26,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28425.20 MB 2025-02-15 03:53:26,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29192.20 MB 2025-02-15 03:53:26,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:53:26,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33709.62 MB 2025-02-15 03:53:26,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34124.86 MB 2025-02-15 03:53:26,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:53:26,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29899.99 MB 2025-02-15 03:53:26,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:53:26,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:53:26,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:53:26,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.09 MB 2025-02-15 03:53:26,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29832.17 MB 2025-02-15 03:53:26,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.08 MB 2025-02-15 03:53:26,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34124.86 MB 2025-02-15 03:53:26,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34124.86 MB 2025-02-15 03:53:26,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:53:26,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30062.73 MB 2025-02-15 03:53:26,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:53:26,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:53:26,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.46 seconds 2025-02-15 03:53:26,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:26,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17292.45 MB 2025-02-15 03:53:26,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30032.80 MB 2025-02-15 03:53:26,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12740.35 MB 2025-02-15 03:53:26,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57984.16 MB 2025-02-15 03:53:26,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34124.86 MB 2025-02-15 03:53:26,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23859.30 MB 2025-02-15 03:53:26,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30062.73 MB 2025-02-15 03:53:27,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:53:27,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:53:27,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:53:27,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:27,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30032.80 MB 2025-02-15 03:53:27,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22290.20 MB 2025-02-15 03:53:27,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7742.60 MB 2025-02-15 03:53:27,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34124.86 MB 2025-02-15 03:53:27,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34124.86 MB 2025-02-15 03:53:27,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:53:27,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32539.16 MB 2025-02-15 03:53:27,231 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 03:53:27,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:53:27,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:53:27,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:53:27,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:53:27,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:53:27,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22290.20 MB 2025-02-15 03:53:27,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30710.31 MB 2025-02-15 03:53:27,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.11 MB 2025-02-15 03:53:27,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34124.86 MB 2025-02-15 03:53:27,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38310.77 MB 2025-02-15 03:53:27,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 03:53:27,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30710.31 MB 2025-02-15 03:53:27,401 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 03:53:27,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:27,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:53:27,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:27,403 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:53:27,408 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:53:27,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:53:27,409 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:53:27,409 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:54:51,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:54:51,975 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:54:51,984 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:54:51,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:54:51,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 363, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:54:51,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:54:51,993 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 363, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:54:57,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:54:57,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:54:57,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.60 seconds 2025-02-15 03:54:57,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:57,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15498.15 MB 2025-02-15 03:54:57,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16782.79 MB 2025-02-15 03:54:57,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1284.64 MB 2025-02-15 03:54:57,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46682.60 MB 2025-02-15 03:54:57,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20669.53 MB 2025-02-15 03:54:57,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26013.07 MB 2025-02-15 03:54:57,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25649.00 MB 2025-02-15 03:54:57,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:54:57,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:54:57,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:54:57,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:57,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16782.79 MB 2025-02-15 03:54:57,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.04 MB 2025-02-15 03:54:57,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 623.26 MB 2025-02-15 03:54:57,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20669.53 MB 2025-02-15 03:54:57,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23863.49 MB 2025-02-15 03:54:57,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3193.96 MB 2025-02-15 03:54:57,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21883.79 MB 2025-02-15 03:54:59,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:54:59,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:54:59,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.71 seconds 2025-02-15 03:54:59,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.04 MB 2025-02-15 03:54:59,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17887.78 MB 2025-02-15 03:54:59,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.74 MB 2025-02-15 03:54:59,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23863.49 MB 2025-02-15 03:54:59,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21955.08 MB 2025-02-15 03:54:59,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1908.41 MB 2025-02-15 03:54:59,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21830.49 MB 2025-02-15 03:54:59,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:54:59,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:54:59,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:54:59,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17887.78 MB 2025-02-15 03:54:59,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19602.73 MB 2025-02-15 03:54:59,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.95 MB 2025-02-15 03:54:59,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21955.08 MB 2025-02-15 03:54:59,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22812.82 MB 2025-02-15 03:54:59,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 857.74 MB 2025-02-15 03:54:59,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20889.04 MB 2025-02-15 03:54:59,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:54:59,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:54:59,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 03:54:59,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19602.73 MB 2025-02-15 03:54:59,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.22 MB 2025-02-15 03:54:59,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2034.49 MB 2025-02-15 03:54:59,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22812.82 MB 2025-02-15 03:54:59,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28819.06 MB 2025-02-15 03:54:59,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6006.24 MB 2025-02-15 03:54:59,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26668.65 MB 2025-02-15 03:54:59,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:54:59,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:54:59,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 03:54:59,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17887.78 MB 2025-02-15 03:54:59,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.22 MB 2025-02-15 03:54:59,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3749.44 MB 2025-02-15 03:54:59,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21955.08 MB 2025-02-15 03:54:59,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28819.06 MB 2025-02-15 03:54:59,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6863.98 MB 2025-02-15 03:54:59,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26668.65 MB 2025-02-15 03:54:59,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:54:59,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:54:59,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 03:54:59,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23028.91 MB 2025-02-15 03:54:59,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23724.96 MB 2025-02-15 03:54:59,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 696.05 MB 2025-02-15 03:54:59,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28819.06 MB 2025-02-15 03:54:59,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29192.36 MB 2025-02-15 03:54:59,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 373.29 MB 2025-02-15 03:54:59,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24367.28 MB 2025-02-15 03:54:59,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:54:59,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:54:59,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:54:59,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24099.66 MB 2025-02-15 03:54:59,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24312.13 MB 2025-02-15 03:54:59,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.47 MB 2025-02-15 03:54:59,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29192.36 MB 2025-02-15 03:54:59,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29192.36 MB 2025-02-15 03:54:59,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:54:59,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24470.18 MB 2025-02-15 03:54:59,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:54:59,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:54:59,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.74 seconds 2025-02-15 03:54:59,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:54:59,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14233.43 MB 2025-02-15 03:54:59,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24513.20 MB 2025-02-15 03:54:59,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10279.77 MB 2025-02-15 03:54:59,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46682.60 MB 2025-02-15 03:54:59,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29192.36 MB 2025-02-15 03:54:59,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17490.25 MB 2025-02-15 03:54:59,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24513.20 MB 2025-02-15 03:55:00,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:55:00,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:55:00,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:55:00,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:55:00,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24513.20 MB 2025-02-15 03:55:00,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19062.81 MB 2025-02-15 03:55:00,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5450.39 MB 2025-02-15 03:55:00,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29192.36 MB 2025-02-15 03:55:00,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29192.36 MB 2025-02-15 03:55:00,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:55:00,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27627.67 MB 2025-02-15 03:55:00,022 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:55:00,023 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:55:00,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:55:00,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:55:00,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:55:00,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:55:00,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19062.81 MB 2025-02-15 03:55:00,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27501.83 MB 2025-02-15 03:55:00,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:55:00,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29192.36 MB 2025-02-15 03:55:00,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39682.31 MB 2025-02-15 03:55:00,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 03:55:00,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27501.83 MB 2025-02-15 03:55:00,187 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:55:00,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:00,189 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:55:00,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:00,190 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:55:00,194 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:55:00,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:00,195 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:55:00,195 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:55:32,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:32,429 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:55:32,437 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:55:32,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:32,444 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1790, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:55:32,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:55:32,445 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1790, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:56:00,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:56:00,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:56:00,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.57 seconds 2025-02-15 03:56:00,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:00,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25441.71 MB 2025-02-15 03:56:00,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31777.21 MB 2025-02-15 03:56:00,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6335.50 MB 2025-02-15 03:56:00,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52267.32 MB 2025-02-15 03:56:00,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39585.84 MB 2025-02-15 03:56:00,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12681.48 MB 2025-02-15 03:56:00,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40575.39 MB 2025-02-15 03:56:00,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:56:00,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:56:00,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 03:56:00,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:00,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31777.21 MB 2025-02-15 03:56:00,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25083.50 MB 2025-02-15 03:56:00,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6693.71 MB 2025-02-15 03:56:00,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39585.84 MB 2025-02-15 03:56:00,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53146.03 MB 2025-02-15 03:56:00,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13560.18 MB 2025-02-15 03:56:00,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50042.69 MB 2025-02-15 03:56:02,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:56:02,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:56:02,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 03:56:02,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25083.50 MB 2025-02-15 03:56:02,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25614.34 MB 2025-02-15 03:56:02,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:56:02,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53146.03 MB 2025-02-15 03:56:02,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30477.91 MB 2025-02-15 03:56:02,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22668.12 MB 2025-02-15 03:56:02,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29592.89 MB 2025-02-15 03:56:02,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:56:02,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:56:02,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:56:02,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-15 03:56:02,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27503.88 MB 2025-02-15 03:56:02,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:56:02,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-15 03:56:02,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30477.91 MB 2025-02-15 03:56:02,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:56:02,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28921.31 MB 2025-02-15 03:56:02,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:56:02,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:56:02,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:56:02,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27503.88 MB 2025-02-15 03:56:02,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-15 03:56:02,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:56:02,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-15 03:56:02,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37083.94 MB 2025-02-15 03:56:02,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:56:02,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-15 03:56:02,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:56:02,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:56:02,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 03:56:02,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-15 03:56:02,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-15 03:56:02,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:56:02,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-15 03:56:02,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37083.94 MB 2025-02-15 03:56:02,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 03:56:02,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-15 03:56:02,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:56:02,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:56:02,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:56:02,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31279.28 MB 2025-02-15 03:56:02,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32046.28 MB 2025-02-15 03:56:02,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:56:02,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37083.94 MB 2025-02-15 03:56:02,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-15 03:56:02,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:56:02,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32754.07 MB 2025-02-15 03:56:02,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:56:02,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:56:02,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:56:02,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32459.17 MB 2025-02-15 03:56:02,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32687.98 MB 2025-02-15 03:56:02,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-15 03:56:02,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37499.17 MB 2025-02-15 03:56:02,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-15 03:56:02,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:56:02,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32897.31 MB 2025-02-15 03:56:02,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:56:02,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:56:02,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.03 seconds 2025-02-15 03:56:02,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19205.21 MB 2025-02-15 03:56:02,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32888.91 MB 2025-02-15 03:56:02,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13683.70 MB 2025-02-15 03:56:02,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52267.32 MB 2025-02-15 03:56:02,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-15 03:56:02,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14768.14 MB 2025-02-15 03:56:02,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32897.31 MB 2025-02-15 03:56:02,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:56:02,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:56:02,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:56:02,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32888.91 MB 2025-02-15 03:56:02,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24207.31 MB 2025-02-15 03:56:02,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8681.59 MB 2025-02-15 03:56:02,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37499.17 MB 2025-02-15 03:56:02,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-15 03:56:02,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:56:02,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35398.73 MB 2025-02-15 03:56:02,771 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 03:56:02,771 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:56:02,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:56:02,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:56:02,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:56:02,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:56:02,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24207.31 MB 2025-02-15 03:56:02,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32640.61 MB 2025-02-15 03:56:02,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 03:56:02,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37499.17 MB 2025-02-15 03:56:02,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41691.38 MB 2025-02-15 03:56:02,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 03:56:02,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.61 MB 2025-02-15 03:56:02,941 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 03:56:02,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:56:02,943 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:56:02,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:56:02,944 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:56:02,949 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:56:02,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:56:02,950 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:56:02,950 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 03:57:05,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:05,673 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:57:05,678 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:57:05,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:05,682 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 640, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:57:05,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:05,683 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 640, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:57:15,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:57:15,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:57:15,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.77 seconds 2025-02-15 03:57:15,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:15,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17428.33 MB 2025-02-15 03:57:15,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19693.25 MB 2025-02-15 03:57:15,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2264.92 MB 2025-02-15 03:57:15,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50075.80 MB 2025-02-15 03:57:15,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24356.32 MB 2025-02-15 03:57:15,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25719.47 MB 2025-02-15 03:57:15,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28711.64 MB 2025-02-15 03:57:15,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:57:15,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:57:15,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 03:57:15,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:15,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19693.25 MB 2025-02-15 03:57:15,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19106.06 MB 2025-02-15 03:57:15,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -587.19 MB 2025-02-15 03:57:15,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24356.32 MB 2025-02-15 03:57:15,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30641.49 MB 2025-02-15 03:57:15,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6285.16 MB 2025-02-15 03:57:15,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28077.77 MB 2025-02-15 03:57:17,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:57:17,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:57:17,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-15 03:57:17,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19106.06 MB 2025-02-15 03:57:17,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19636.90 MB 2025-02-15 03:57:17,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:57:17,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30641.49 MB 2025-02-15 03:57:17,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24215.81 MB 2025-02-15 03:57:17,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6425.67 MB 2025-02-15 03:57:17,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23615.45 MB 2025-02-15 03:57:17,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:57:17,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:57:17,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:57:17,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19636.90 MB 2025-02-15 03:57:17,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21526.43 MB 2025-02-15 03:57:17,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:57:17,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24215.81 MB 2025-02-15 03:57:17,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25159.53 MB 2025-02-15 03:57:17,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:57:17,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22943.86 MB 2025-02-15 03:57:17,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:57:17,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:57:17,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:57:17,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21526.43 MB 2025-02-15 03:57:17,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23769.34 MB 2025-02-15 03:57:17,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 03:57:17,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25159.53 MB 2025-02-15 03:57:17,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31530.68 MB 2025-02-15 03:57:17,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 03:57:17,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29313.62 MB 2025-02-15 03:57:17,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:57:17,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:57:17,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:57:17,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19636.90 MB 2025-02-15 03:57:17,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23769.34 MB 2025-02-15 03:57:17,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 03:57:17,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24215.81 MB 2025-02-15 03:57:17,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31530.68 MB 2025-02-15 03:57:17,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 03:57:17,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29313.62 MB 2025-02-15 03:57:17,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:57:17,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:57:17,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:57:17,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25302.88 MB 2025-02-15 03:57:17,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26069.88 MB 2025-02-15 03:57:17,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:57:17,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31530.68 MB 2025-02-15 03:57:17,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 03:57:17,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:57:17,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26777.67 MB 2025-02-15 03:57:17,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:57:17,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:57:17,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:57:17,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26482.77 MB 2025-02-15 03:57:17,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26712.72 MB 2025-02-15 03:57:17,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.94 MB 2025-02-15 03:57:17,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31943.82 MB 2025-02-15 03:57:17,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 03:57:17,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:57:17,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26900.07 MB 2025-02-15 03:57:17,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:57:17,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:57:17,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.12 seconds 2025-02-15 03:57:17,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:17,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15198.52 MB 2025-02-15 03:57:17,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26913.79 MB 2025-02-15 03:57:17,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11715.27 MB 2025-02-15 03:57:17,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50075.80 MB 2025-02-15 03:57:17,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 03:57:17,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18131.98 MB 2025-02-15 03:57:17,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26913.79 MB 2025-02-15 03:57:18,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:57:18,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:57:18,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:57:18,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:18,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26913.79 MB 2025-02-15 03:57:18,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20202.91 MB 2025-02-15 03:57:18,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6710.88 MB 2025-02-15 03:57:18,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31943.82 MB 2025-02-15 03:57:18,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 03:57:18,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:57:18,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29425.46 MB 2025-02-15 03:57:18,087 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 03:57:18,088 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:57:18,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:57:18,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:57:18,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:57:18,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:57:18,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20202.91 MB 2025-02-15 03:57:18,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28641.93 MB 2025-02-15 03:57:18,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 03:57:18,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31943.82 MB 2025-02-15 03:57:18,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40334.52 MB 2025-02-15 03:57:18,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 03:57:18,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28641.93 MB 2025-02-15 03:57:18,252 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 03:57:18,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:18,253 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:57:18,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:18,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:57:18,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:57:18,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:57:18,260 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:57:18,260 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 03:58:14,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:14,149 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:58:14,157 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:58:14,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:14,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:58:14,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:14,167 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:58:33,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:58:33,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:58:33,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.78 seconds 2025-02-15 03:58:33,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:33,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-15 03:58:33,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-15 03:58:33,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-15 03:58:33,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52919.53 MB 2025-02-15 03:58:33,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37807.46 MB 2025-02-15 03:58:33,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15112.08 MB 2025-02-15 03:58:33,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.03 MB 2025-02-15 03:58:34,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:58:34,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:58:34,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 03:58:34,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:34,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-15 03:58:34,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-15 03:58:34,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-15 03:58:34,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37807.46 MB 2025-02-15 03:58:34,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46506.44 MB 2025-02-15 03:58:34,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8698.99 MB 2025-02-15 03:58:34,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39547.05 MB 2025-02-15 03:58:35,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:58:35,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:58:35,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 03:58:35,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:35,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-15 03:58:35,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-15 03:58:35,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:58:35,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46506.44 MB 2025-02-15 03:58:35,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 03:58:35,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13247.71 MB 2025-02-15 03:58:35,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26967.55 MB 2025-02-15 03:58:35,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:58:35,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:58:35,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:58:35,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:35,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-15 03:58:35,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-15 03:58:35,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:58:35,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 03:58:35,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 03:58:35,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:58:35,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-15 03:58:36,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:58:36,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:58:36,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:58:36,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-15 03:58:36,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-15 03:58:36,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 03:58:36,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 03:58:36,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34202.45 MB 2025-02-15 03:58:36,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:58:36,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-15 03:58:36,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:58:36,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:58:36,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:58:36,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-15 03:58:36,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-15 03:58:36,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 03:58:36,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 03:58:36,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34202.45 MB 2025-02-15 03:58:36,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:58:36,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-15 03:58:36,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:58:36,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:58:36,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:58:36,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-15 03:58:36,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-15 03:58:36,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:58:36,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34202.45 MB 2025-02-15 03:58:36,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34615.59 MB 2025-02-15 03:58:36,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 03:58:36,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-15 03:58:36,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:58:36,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:58:36,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:58:36,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-15 03:58:36,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30061.91 MB 2025-02-15 03:58:36,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-15 03:58:36,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34615.59 MB 2025-02-15 03:58:36,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34615.59 MB 2025-02-15 03:58:36,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:58:36,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30277.05 MB 2025-02-15 03:58:36,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:58:36,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:58:36,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.17 seconds 2025-02-15 03:58:36,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-15 03:58:36,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30262.76 MB 2025-02-15 03:58:36,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12817.01 MB 2025-02-15 03:58:36,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52919.53 MB 2025-02-15 03:58:36,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34615.59 MB 2025-02-15 03:58:36,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18303.94 MB 2025-02-15 03:58:36,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30277.05 MB 2025-02-15 03:58:36,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:58:36,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:58:36,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 03:58:36,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30262.76 MB 2025-02-15 03:58:36,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22434.24 MB 2025-02-15 03:58:36,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7828.52 MB 2025-02-15 03:58:36,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34615.59 MB 2025-02-15 03:58:36,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34615.59 MB 2025-02-15 03:58:36,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:58:36,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32760.91 MB 2025-02-15 03:58:36,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-15 03:58:36,627 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:58:36,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:58:36,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:58:36,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:58:36,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:58:36,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22434.24 MB 2025-02-15 03:58:36,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30827.51 MB 2025-02-15 03:58:36,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-15 03:58:36,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34615.59 MB 2025-02-15 03:58:36,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38788.92 MB 2025-02-15 03:58:36,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 03:58:36,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30827.51 MB 2025-02-15 03:58:36,791 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-15 03:58:36,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:36,793 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:58:36,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:36,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:58:36,798 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:58:36,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:36,799 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:58:36,800 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 03:58:51,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:51,714 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 03:58:51,719 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 03:58:51,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:51,722 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 03:58:51,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:58:51,723 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 03:59:10,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 03:59:10,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 03:59:10,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.39 seconds 2025-02-15 03:59:10,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:10,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21246.88 MB 2025-02-15 03:59:10,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25451.67 MB 2025-02-15 03:59:10,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4204.79 MB 2025-02-15 03:59:10,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47135.59 MB 2025-02-15 03:59:10,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29051.85 MB 2025-02-15 03:59:10,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18083.74 MB 2025-02-15 03:59:10,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34342.48 MB 2025-02-15 03:59:10,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 03:59:10,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 03:59:10,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 03:59:10,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:10,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25451.67 MB 2025-02-15 03:59:10,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21954.94 MB 2025-02-15 03:59:10,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3496.73 MB 2025-02-15 03:59:10,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29051.85 MB 2025-02-15 03:59:10,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39508.25 MB 2025-02-15 03:59:10,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10456.40 MB 2025-02-15 03:59:10,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38008.18 MB 2025-02-15 03:59:12,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 03:59:12,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 03:59:12,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 03:59:12,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21954.94 MB 2025-02-15 03:59:12,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22485.78 MB 2025-02-15 03:59:12,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 03:59:12,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39508.25 MB 2025-02-15 03:59:12,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26971.47 MB 2025-02-15 03:59:12,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12536.77 MB 2025-02-15 03:59:12,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26466.41 MB 2025-02-15 03:59:12,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 03:59:12,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 03:59:12,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 03:59:12,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22485.78 MB 2025-02-15 03:59:12,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24375.32 MB 2025-02-15 03:59:12,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 03:59:12,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-15 03:59:12,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27915.19 MB 2025-02-15 03:59:12,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 03:59:12,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25792.75 MB 2025-02-15 03:59:12,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 03:59:12,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 03:59:12,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 03:59:12,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24375.32 MB 2025-02-15 03:59:12,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26618.22 MB 2025-02-15 03:59:12,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 03:59:12,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27915.19 MB 2025-02-15 03:59:12,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-15 03:59:12,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 03:59:12,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32162.50 MB 2025-02-15 03:59:12,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 03:59:12,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 03:59:12,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 03:59:12,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22485.78 MB 2025-02-15 03:59:12,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26618.22 MB 2025-02-15 03:59:12,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 03:59:12,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-15 03:59:12,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-15 03:59:12,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 03:59:12,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32162.50 MB 2025-02-15 03:59:12,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 03:59:12,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 03:59:12,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 03:59:12,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28151.76 MB 2025-02-15 03:59:12,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28918.77 MB 2025-02-15 03:59:12,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 03:59:12,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-15 03:59:12,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:59:12,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 03:59:12,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29626.55 MB 2025-02-15 03:59:12,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 03:59:12,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 03:59:12,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:59:12,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29331.65 MB 2025-02-15 03:59:12,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29559.80 MB 2025-02-15 03:59:12,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-15 03:59:12,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-15 03:59:12,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:59:12,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:59:12,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29795.15 MB 2025-02-15 03:59:12,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 03:59:12,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 03:59:12,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.83 seconds 2025-02-15 03:59:12,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17107.79 MB 2025-02-15 03:59:12,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29760.66 MB 2025-02-15 03:59:12,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12652.86 MB 2025-02-15 03:59:12,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47135.59 MB 2025-02-15 03:59:12,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:59:12,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12434.01 MB 2025-02-15 03:59:12,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29795.15 MB 2025-02-15 03:59:12,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 03:59:12,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 03:59:12,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 03:59:12,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29760.66 MB 2025-02-15 03:59:12,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22097.35 MB 2025-02-15 03:59:12,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7663.30 MB 2025-02-15 03:59:12,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-15 03:59:12,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-15 03:59:12,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 03:59:12,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32259.73 MB 2025-02-15 03:59:12,844 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 03:59:12,844 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 03:59:12,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 03:59:12,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 03:59:12,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 03:59:12,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 03:59:12,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22097.35 MB 2025-02-15 03:59:12,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30494.12 MB 2025-02-15 03:59:12,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-15 03:59:12,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-15 03:59:12,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43050.34 MB 2025-02-15 03:59:12,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-15 03:59:12,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30494.12 MB 2025-02-15 03:59:13,007 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 03:59:13,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:59:13,009 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 03:59:13,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:59:13,010 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 03:59:13,014 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 03:59:13,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 03:59:13,015 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 03:59:13,015 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:00:14,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:14,427 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:00:14,431 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:00:14,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:14,435 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 275, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:00:14,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:14,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 275, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:00:18,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:00:18,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:00:18,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.24 seconds 2025-02-15 04:00:18,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:18,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14885.29 MB 2025-02-15 04:00:18,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15858.50 MB 2025-02-15 04:00:18,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 973.21 MB 2025-02-15 04:00:18,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55572.43 MB 2025-02-15 04:00:18,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18601.74 MB 2025-02-15 04:00:18,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36970.69 MB 2025-02-15 04:00:18,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.65 MB 2025-02-15 04:00:18,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:00:18,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:00:18,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:00:18,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:18,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15858.50 MB 2025-02-15 04:00:18,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16316.56 MB 2025-02-15 04:00:18,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.06 MB 2025-02-15 04:00:18,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18601.74 MB 2025-02-15 04:00:18,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21474.84 MB 2025-02-15 04:00:18,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2873.10 MB 2025-02-15 04:00:18,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19694.38 MB 2025-02-15 04:00:20,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:00:20,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:00:20,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.30 seconds 2025-02-15 04:00:20,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16316.56 MB 2025-02-15 04:00:20,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16678.86 MB 2025-02-15 04:00:20,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 362.30 MB 2025-02-15 04:00:20,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21474.84 MB 2025-02-15 04:00:20,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20038.29 MB 2025-02-15 04:00:20,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1436.55 MB 2025-02-15 04:00:20,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20657.12 MB 2025-02-15 04:00:20,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:00:20,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:00:20,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:00:20,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16678.86 MB 2025-02-15 04:00:20,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17968.18 MB 2025-02-15 04:00:20,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1289.31 MB 2025-02-15 04:00:20,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20038.29 MB 2025-02-15 04:00:20,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20038.29 MB 2025-02-15 04:00:20,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:00:20,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18935.58 MB 2025-02-15 04:00:20,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:00:20,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:00:20,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:00:20,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17968.18 MB 2025-02-15 04:00:20,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19499.18 MB 2025-02-15 04:00:20,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1531.01 MB 2025-02-15 04:00:20,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20038.29 MB 2025-02-15 04:00:20,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24882.71 MB 2025-02-15 04:00:20,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4844.42 MB 2025-02-15 04:00:20,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23284.05 MB 2025-02-15 04:00:20,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:00:20,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:00:20,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 04:00:20,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16678.86 MB 2025-02-15 04:00:20,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19499.18 MB 2025-02-15 04:00:20,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2820.32 MB 2025-02-15 04:00:20,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20038.29 MB 2025-02-15 04:00:20,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24882.71 MB 2025-02-15 04:00:20,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4844.42 MB 2025-02-15 04:00:20,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23284.05 MB 2025-02-15 04:00:20,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:00:20,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:00:20,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:00:20,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20545.83 MB 2025-02-15 04:00:20,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21071.14 MB 2025-02-15 04:00:20,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 525.31 MB 2025-02-15 04:00:20,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24882.71 MB 2025-02-15 04:00:20,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-15 04:00:20,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 283.12 MB 2025-02-15 04:00:20,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21554.21 MB 2025-02-15 04:00:20,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:00:20,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:00:20,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:00:20,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21352.94 MB 2025-02-15 04:00:20,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21580.86 MB 2025-02-15 04:00:20,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-15 04:00:20,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-15 04:00:20,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-15 04:00:20,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:00:20,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21703.66 MB 2025-02-15 04:00:20,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:00:20,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:00:20,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.85 seconds 2025-02-15 04:00:20,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13926.83 MB 2025-02-15 04:00:20,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21781.93 MB 2025-02-15 04:00:20,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7855.10 MB 2025-02-15 04:00:20,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55572.43 MB 2025-02-15 04:00:20,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-15 04:00:20,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30406.61 MB 2025-02-15 04:00:20,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21781.93 MB 2025-02-15 04:00:20,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:00:20,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:00:20,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:00:20,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21781.93 MB 2025-02-15 04:00:20,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24795.97 MB 2025-02-15 04:00:20,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:00:20,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-15 04:00:20,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26373.78 MB 2025-02-15 04:00:20,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-15 04:00:20,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25097.59 MB 2025-02-15 04:00:20,572 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:00:20,572 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:00:20,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:00:20,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:00:20,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:00:20,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:00:20,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18333.28 MB 2025-02-15 04:00:20,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26772.30 MB 2025-02-15 04:00:20,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:00:20,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26373.78 MB 2025-02-15 04:00:20,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36863.74 MB 2025-02-15 04:00:20,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:00:20,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26772.30 MB 2025-02-15 04:00:20,736 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:00:20,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:20,738 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:00:20,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:20,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:00:20,743 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:00:20,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:00:20,744 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:00:20,744 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:01:06,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:06,293 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:01:06,297 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:01:06,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:06,301 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1406, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:01:06,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:06,302 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1406, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:01:27,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:01:27,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:01:27,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.53 seconds 2025-02-15 04:01:27,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:27,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22765.94 MB 2025-02-15 04:01:27,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27742.48 MB 2025-02-15 04:01:27,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4976.54 MB 2025-02-15 04:01:27,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49448.75 MB 2025-02-15 04:01:27,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38237.37 MB 2025-02-15 04:01:27,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11211.37 MB 2025-02-15 04:01:27,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36540.67 MB 2025-02-15 04:01:27,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:01:27,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:01:27,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:01:27,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:27,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27742.48 MB 2025-02-15 04:01:27,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23087.21 MB 2025-02-15 04:01:27,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4655.28 MB 2025-02-15 04:01:27,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38237.37 MB 2025-02-15 04:01:27,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48039.46 MB 2025-02-15 04:01:27,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9802.09 MB 2025-02-15 04:01:27,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42477.44 MB 2025-02-15 04:01:29,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:01:29,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:01:29,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:01:29,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:29,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23087.21 MB 2025-02-15 04:01:29,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23618.05 MB 2025-02-15 04:01:29,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:01:29,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48039.46 MB 2025-02-15 04:01:29,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29066.53 MB 2025-02-15 04:01:29,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18972.93 MB 2025-02-15 04:01:29,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27596.60 MB 2025-02-15 04:01:29,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:01:29,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:01:29,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:01:29,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:29,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23618.05 MB 2025-02-15 04:01:29,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25507.58 MB 2025-02-15 04:01:29,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:01:29,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 04:01:29,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30010.25 MB 2025-02-15 04:01:29,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:01:29,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26925.01 MB 2025-02-15 04:01:30,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:01:30,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:01:30,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:01:30,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25507.58 MB 2025-02-15 04:01:30,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27749.44 MB 2025-02-15 04:01:30,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:01:30,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30010.25 MB 2025-02-15 04:01:30,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 04:01:30,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:01:30,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33293.72 MB 2025-02-15 04:01:30,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:01:30,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:01:30,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:01:30,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23618.05 MB 2025-02-15 04:01:30,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27749.44 MB 2025-02-15 04:01:30,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:01:30,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 04:01:30,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 04:01:30,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:01:30,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33293.72 MB 2025-02-15 04:01:30,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:01:30,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:01:30,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:01:30,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29282.98 MB 2025-02-15 04:01:30,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30049.98 MB 2025-02-15 04:01:30,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:01:30,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35672.56 MB 2025-02-15 04:01:30,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:01:30,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:01:30,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30757.77 MB 2025-02-15 04:01:30,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:01:30,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:01:30,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:01:30,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30462.87 MB 2025-02-15 04:01:30,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30690.26 MB 2025-02-15 04:01:30,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.39 MB 2025-02-15 04:01:30,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:01:30,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:01:30,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:01:30,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30930.51 MB 2025-02-15 04:01:30,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:01:30,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:01:30,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.94 seconds 2025-02-15 04:01:30,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17867.32 MB 2025-02-15 04:01:30,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30890.27 MB 2025-02-15 04:01:30,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13022.95 MB 2025-02-15 04:01:30,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49448.75 MB 2025-02-15 04:01:30,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:01:30,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13363.05 MB 2025-02-15 04:01:30,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30930.51 MB 2025-02-15 04:01:30,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:01:30,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:01:30,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:01:30,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30890.27 MB 2025-02-15 04:01:30,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22856.17 MB 2025-02-15 04:01:30,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8034.11 MB 2025-02-15 04:01:30,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:01:30,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:01:30,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:01:30,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33388.73 MB 2025-02-15 04:01:30,527 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 04:01:30,527 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:01:30,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:01:30,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:01:30,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:01:30,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:30,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22856.17 MB 2025-02-15 04:01:30,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31251.38 MB 2025-02-15 04:01:30,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 04:01:30,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:01:30,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44432.36 MB 2025-02-15 04:01:30,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 04:01:30,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31251.38 MB 2025-02-15 04:01:30,692 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 04:01:30,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:30,693 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:01:30,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:30,694 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:01:30,699 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:01:30,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:30,700 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:01:30,700 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:01:42,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:42,063 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:01:42,068 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:01:42,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:42,072 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 932, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:01:42,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:42,073 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 932, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:01:56,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:01:56,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:01:56,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.53 seconds 2025-02-15 04:01:56,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:56,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19463.03 MB 2025-02-15 04:01:56,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22761.85 MB 2025-02-15 04:01:56,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3298.82 MB 2025-02-15 04:01:56,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52779.02 MB 2025-02-15 04:01:56,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28166.85 MB 2025-02-15 04:01:56,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24612.18 MB 2025-02-15 04:01:56,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31652.31 MB 2025-02-15 04:01:56,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:01:56,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:01:56,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:01:56,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:56,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22761.85 MB 2025-02-15 04:01:56,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20623.03 MB 2025-02-15 04:01:56,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2138.82 MB 2025-02-15 04:01:56,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28166.85 MB 2025-02-15 04:01:56,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35626.42 MB 2025-02-15 04:01:56,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7459.57 MB 2025-02-15 04:01:56,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33027.22 MB 2025-02-15 04:01:58,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:01:58,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:01:58,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:01:58,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:58,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20623.03 MB 2025-02-15 04:01:58,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21153.87 MB 2025-02-15 04:01:58,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:01:58,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35626.42 MB 2025-02-15 04:01:58,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 04:01:58,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9342.81 MB 2025-02-15 04:01:58,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25133.07 MB 2025-02-15 04:01:58,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:01:58,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:01:58,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:01:58,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:58,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21153.87 MB 2025-02-15 04:01:58,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23043.40 MB 2025-02-15 04:01:58,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:01:58,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 04:01:58,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 04:01:58,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:01:58,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24460.83 MB 2025-02-15 04:01:58,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:01:58,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:01:58,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:01:58,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:58,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23043.40 MB 2025-02-15 04:01:58,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25285.26 MB 2025-02-15 04:01:58,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:01:58,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 04:01:58,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32889.63 MB 2025-02-15 04:01:58,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:01:58,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30829.54 MB 2025-02-15 04:01:58,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:01:58,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:01:58,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:01:58,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:58,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21153.87 MB 2025-02-15 04:01:58,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25285.26 MB 2025-02-15 04:01:58,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:01:58,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 04:01:58,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32889.63 MB 2025-02-15 04:01:58,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:01:58,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30829.54 MB 2025-02-15 04:01:58,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:01:58,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:01:58,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:01:58,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:58,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26818.80 MB 2025-02-15 04:01:58,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27585.80 MB 2025-02-15 04:01:58,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:01:58,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32889.63 MB 2025-02-15 04:01:58,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33304.87 MB 2025-02-15 04:01:58,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:01:58,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28293.59 MB 2025-02-15 04:01:59,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:01:59,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:01:59,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:01:59,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:59,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27998.69 MB 2025-02-15 04:01:59,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28228.49 MB 2025-02-15 04:01:59,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.80 MB 2025-02-15 04:01:59,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33304.87 MB 2025-02-15 04:01:59,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33304.87 MB 2025-02-15 04:01:59,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:01:59,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28442.02 MB 2025-02-15 04:01:59,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:01:59,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:01:59,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.94 seconds 2025-02-15 04:01:59,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:59,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16215.87 MB 2025-02-15 04:01:59,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28429.44 MB 2025-02-15 04:01:59,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12213.57 MB 2025-02-15 04:01:59,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52779.02 MB 2025-02-15 04:01:59,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33304.87 MB 2025-02-15 04:01:59,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19474.15 MB 2025-02-15 04:01:59,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28442.02 MB 2025-02-15 04:01:59,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:01:59,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:01:59,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:01:59,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:59,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28429.44 MB 2025-02-15 04:01:59,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21218.35 MB 2025-02-15 04:01:59,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7211.09 MB 2025-02-15 04:01:59,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33304.87 MB 2025-02-15 04:01:59,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33304.87 MB 2025-02-15 04:01:59,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:01:59,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30939.57 MB 2025-02-15 04:01:59,301 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 04:01:59,302 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:01:59,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:01:59,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:01:59,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:01:59,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:01:59,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21218.35 MB 2025-02-15 04:01:59,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29652.97 MB 2025-02-15 04:01:59,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 04:01:59,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33304.87 MB 2025-02-15 04:01:59,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41689.28 MB 2025-02-15 04:01:59,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 04:01:59,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29652.97 MB 2025-02-15 04:01:59,469 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 04:01:59,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:59,470 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:01:59,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:59,471 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:01:59,476 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:01:59,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:01:59,477 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:01:59,477 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:03:22,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:22,142 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:03:22,147 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:03:22,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:22,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:03:22,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:22,152 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:03:25,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:03:25,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:03:25,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.64 seconds 2025-02-15 04:03:25,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:25,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14627.13 MB 2025-02-15 04:03:25,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15469.40 MB 2025-02-15 04:03:25,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 842.27 MB 2025-02-15 04:03:25,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50073.70 MB 2025-02-15 04:03:25,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21619.54 MB 2025-02-15 04:03:25,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28454.16 MB 2025-02-15 04:03:25,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24324.99 MB 2025-02-15 04:03:25,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:03:25,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:03:25,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:03:25,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:25,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15469.40 MB 2025-02-15 04:03:25,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15694.81 MB 2025-02-15 04:03:25,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.41 MB 2025-02-15 04:03:25,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21619.54 MB 2025-02-15 04:03:25,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21619.54 MB 2025-02-15 04:03:25,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:03:25,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18447.16 MB 2025-02-15 04:03:26,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:03:26,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:03:26,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-15 04:03:26,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:26,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15694.81 MB 2025-02-15 04:03:26,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15976.16 MB 2025-02-15 04:03:26,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.35 MB 2025-02-15 04:03:26,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21619.54 MB 2025-02-15 04:03:26,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 04:03:26,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 04:03:26,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19949.39 MB 2025-02-15 04:03:26,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:03:26,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:03:26,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:03:26,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:26,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15976.16 MB 2025-02-15 04:03:26,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16977.37 MB 2025-02-15 04:03:26,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.21 MB 2025-02-15 04:03:26,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 04:03:26,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 04:03:26,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:03:26,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17728.61 MB 2025-02-15 04:03:26,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:03:26,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:03:26,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:03:26,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:26,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16977.37 MB 2025-02-15 04:03:26,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18165.58 MB 2025-02-15 04:03:26,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1188.21 MB 2025-02-15 04:03:26,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 04:03:26,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22651.34 MB 2025-02-15 04:03:26,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1503.66 MB 2025-02-15 04:03:26,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21107.16 MB 2025-02-15 04:03:26,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:03:26,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:03:26,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 04:03:26,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:26,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15976.16 MB 2025-02-15 04:03:26,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18165.58 MB 2025-02-15 04:03:26,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2189.42 MB 2025-02-15 04:03:26,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 04:03:26,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22651.34 MB 2025-02-15 04:03:26,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1503.66 MB 2025-02-15 04:03:26,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21107.16 MB 2025-02-15 04:03:27,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:03:27,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:03:27,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:03:27,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:27,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18978.36 MB 2025-02-15 04:03:27,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19384.87 MB 2025-02-15 04:03:27,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.51 MB 2025-02-15 04:03:27,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22651.34 MB 2025-02-15 04:03:27,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22867.35 MB 2025-02-15 04:03:27,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-15 04:03:27,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19760.64 MB 2025-02-15 04:03:27,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:03:27,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:03:27,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:03:27,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:27,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19603.71 MB 2025-02-15 04:03:27,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19814.92 MB 2025-02-15 04:03:27,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.22 MB 2025-02-15 04:03:27,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22867.35 MB 2025-02-15 04:03:27,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22867.35 MB 2025-02-15 04:03:27,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:03:27,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19831.93 MB 2025-02-15 04:03:27,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:03:27,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:03:27,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.89 seconds 2025-02-15 04:03:27,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:27,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13797.92 MB 2025-02-15 04:03:27,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20015.41 MB 2025-02-15 04:03:27,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6217.49 MB 2025-02-15 04:03:27,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50073.70 MB 2025-02-15 04:03:27,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22867.35 MB 2025-02-15 04:03:27,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27206.35 MB 2025-02-15 04:03:27,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20015.41 MB 2025-02-15 04:03:27,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:03:27,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:03:27,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:03:27,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:27,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14900.75 MB 2025-02-15 04:03:27,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17906.30 MB 2025-02-15 04:03:27,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-15 04:03:27,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22867.35 MB 2025-02-15 04:03:27,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22867.35 MB 2025-02-15 04:03:27,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:03:27,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18206.78 MB 2025-02-15 04:03:27,327 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 04:03:27,327 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:03:27,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:03:27,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:03:27,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:03:27,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:03:27,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17906.30 MB 2025-02-15 04:03:27,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26320.28 MB 2025-02-15 04:03:27,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 04:03:27,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22867.35 MB 2025-02-15 04:03:27,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33325.84 MB 2025-02-15 04:03:27,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10458.50 MB 2025-02-15 04:03:27,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26320.28 MB 2025-02-15 04:03:27,498 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 04:03:27,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:27,499 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:03:27,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:27,500 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:03:27,505 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:03:27,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:27,506 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:03:27,506 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:03:54,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:54,557 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:03:54,561 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:03:54,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:54,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1782, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:03:54,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:03:54,566 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1782, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:04:21,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:04:21,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:04:21,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.36 seconds 2025-02-15 04:04:21,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:21,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25385.97 MB 2025-02-15 04:04:21,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31692.37 MB 2025-02-15 04:04:21,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6306.40 MB 2025-02-15 04:04:21,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45873.10 MB 2025-02-15 04:04:21,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39516.64 MB 2025-02-15 04:04:21,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6356.47 MB 2025-02-15 04:04:21,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40519.65 MB 2025-02-15 04:04:22,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:04:22,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:04:22,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:04:22,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:22,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31692.37 MB 2025-02-15 04:04:22,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25041.91 MB 2025-02-15 04:04:22,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6650.45 MB 2025-02-15 04:04:22,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39516.64 MB 2025-02-15 04:04:22,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 04:04:22,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13673.43 MB 2025-02-15 04:04:22,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50126.77 MB 2025-02-15 04:04:23,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:04:23,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:04:23,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 04:04:23,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:23,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25041.91 MB 2025-02-15 04:04:23,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25572.75 MB 2025-02-15 04:04:23,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:04:23,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53190.07 MB 2025-02-15 04:04:23,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34623.98 MB 2025-02-15 04:04:23,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18566.09 MB 2025-02-15 04:04:23,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29551.30 MB 2025-02-15 04:04:23,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:04:23,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:04:23,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:04:23,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:23,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.75 MB 2025-02-15 04:04:23,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27462.29 MB 2025-02-15 04:04:23,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:04:23,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34623.98 MB 2025-02-15 04:04:23,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34623.98 MB 2025-02-15 04:04:23,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:04:23,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28879.72 MB 2025-02-15 04:04:24,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:04:24,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:04:24,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:04:24,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27462.29 MB 2025-02-15 04:04:24,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29704.14 MB 2025-02-15 04:04:24,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:04:24,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34623.98 MB 2025-02-15 04:04:24,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37455.13 MB 2025-02-15 04:04:24,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:04:24,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35248.43 MB 2025-02-15 04:04:24,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:04:24,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:04:24,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:04:24,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.75 MB 2025-02-15 04:04:24,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29704.14 MB 2025-02-15 04:04:24,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:04:24,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34623.98 MB 2025-02-15 04:04:24,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37455.13 MB 2025-02-15 04:04:24,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:04:24,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35248.43 MB 2025-02-15 04:04:24,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:04:24,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:04:24,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:04:24,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31237.69 MB 2025-02-15 04:04:24,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32004.69 MB 2025-02-15 04:04:24,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:04:24,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37455.13 MB 2025-02-15 04:04:24,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37870.37 MB 2025-02-15 04:04:24,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:04:24,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32712.48 MB 2025-02-15 04:04:24,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:04:24,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:04:24,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:04:24,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32417.58 MB 2025-02-15 04:04:24,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32645.95 MB 2025-02-15 04:04:24,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 04:04:24,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37870.37 MB 2025-02-15 04:04:24,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37870.37 MB 2025-02-15 04:04:24,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:04:24,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32856.46 MB 2025-02-15 04:04:24,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:04:24,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:04:24,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.82 seconds 2025-02-15 04:04:24,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19177.34 MB 2025-02-15 04:04:24,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32846.65 MB 2025-02-15 04:04:24,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13669.32 MB 2025-02-15 04:04:24,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45873.10 MB 2025-02-15 04:04:24,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37870.37 MB 2025-02-15 04:04:24,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8002.73 MB 2025-02-15 04:04:24,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32856.46 MB 2025-02-15 04:04:24,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:04:24,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:04:24,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:04:24,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32846.65 MB 2025-02-15 04:04:24,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24176.16 MB 2025-02-15 04:04:24,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8670.49 MB 2025-02-15 04:04:24,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37870.37 MB 2025-02-15 04:04:24,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37870.37 MB 2025-02-15 04:04:24,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:04:24,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35353.86 MB 2025-02-15 04:04:24,670 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 04:04:24,671 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:04:24,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:04:24,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:04:24,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:04:24,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:24,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24176.16 MB 2025-02-15 04:04:24,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32599.37 MB 2025-02-15 04:04:24,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 04:04:24,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37870.37 MB 2025-02-15 04:04:24,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-15 04:04:24,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 04:04:24,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32599.37 MB 2025-02-15 04:04:24,834 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 04:04:24,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:24,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:04:24,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:24,836 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:04:24,841 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:04:24,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:24,842 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:04:24,842 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:04:35,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:35,557 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:04:35,562 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:04:35,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:35,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 529, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:04:35,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:35,566 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 529, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:04:43,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:04:43,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:04:43,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.26 seconds 2025-02-15 04:04:43,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:43,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16654.86 MB 2025-02-15 04:04:43,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18527.62 MB 2025-02-15 04:04:43,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1872.76 MB 2025-02-15 04:04:43,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54622.42 MB 2025-02-15 04:04:43,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20241.71 MB 2025-02-15 04:04:43,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34380.71 MB 2025-02-15 04:04:43,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27486.00 MB 2025-02-15 04:04:43,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:04:43,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:04:43,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:04:43,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:43,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18527.62 MB 2025-02-15 04:04:43,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18529.01 MB 2025-02-15 04:04:43,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1.39 MB 2025-02-15 04:04:43,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20241.71 MB 2025-02-15 04:04:43,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25971.13 MB 2025-02-15 04:04:43,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5729.42 MB 2025-02-15 04:04:43,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26357.25 MB 2025-02-15 04:04:45,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:04:45,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:04:45,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 04:04:45,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:45,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18529.01 MB 2025-02-15 04:04:45,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19059.85 MB 2025-02-15 04:04:45,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:04:45,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25971.13 MB 2025-02-15 04:04:45,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20082.33 MB 2025-02-15 04:04:45,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5888.80 MB 2025-02-15 04:04:45,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23042.50 MB 2025-02-15 04:04:45,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:04:45,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:04:45,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:04:45,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:45,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19059.85 MB 2025-02-15 04:04:45,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20949.12 MB 2025-02-15 04:04:45,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 04:04:45,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20082.33 MB 2025-02-15 04:04:45,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23857.20 MB 2025-02-15 04:04:45,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 04:04:45,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22366.55 MB 2025-02-15 04:04:46,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:04:46,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:04:46,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:04:46,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20949.12 MB 2025-02-15 04:04:46,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23190.97 MB 2025-02-15 04:04:46,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:04:46,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23857.20 MB 2025-02-15 04:04:46,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-15 04:04:46,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:04:46,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28735.26 MB 2025-02-15 04:04:46,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:04:46,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:04:46,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:04:46,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19059.85 MB 2025-02-15 04:04:46,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23190.97 MB 2025-02-15 04:04:46,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 04:04:46,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20082.33 MB 2025-02-15 04:04:46,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-15 04:04:46,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10380.90 MB 2025-02-15 04:04:46,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28735.26 MB 2025-02-15 04:04:46,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:04:46,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:04:46,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:04:46,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24724.52 MB 2025-02-15 04:04:46,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25491.52 MB 2025-02-15 04:04:46,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:04:46,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 04:04:46,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-15 04:04:46,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:04:46,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26199.31 MB 2025-02-15 04:04:46,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:04:46,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:04:46,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:04:46,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25904.41 MB 2025-02-15 04:04:46,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26133.97 MB 2025-02-15 04:04:46,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.56 MB 2025-02-15 04:04:46,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-15 04:04:46,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-15 04:04:46,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:04:46,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26354.64 MB 2025-02-15 04:04:46,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:04:46,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:04:46,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.66 seconds 2025-02-15 04:04:46,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14811.78 MB 2025-02-15 04:04:46,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26335.04 MB 2025-02-15 04:04:46,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11523.26 MB 2025-02-15 04:04:46,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54622.42 MB 2025-02-15 04:04:46,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-15 04:04:46,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23746.05 MB 2025-02-15 04:04:46,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26354.64 MB 2025-02-15 04:04:46,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:04:46,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:04:46,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:04:46,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26335.04 MB 2025-02-15 04:04:46,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19816.17 MB 2025-02-15 04:04:46,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6518.87 MB 2025-02-15 04:04:46,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-15 04:04:46,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-15 04:04:46,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:04:46,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28846.71 MB 2025-02-15 04:04:46,522 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:04:46,522 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:04:46,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:04:46,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:04:46,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:04:46,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:04:46,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19816.17 MB 2025-02-15 04:04:46,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28255.20 MB 2025-02-15 04:04:46,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:04:46,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-15 04:04:46,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41366.32 MB 2025-02-15 04:04:46,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:04:46,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28255.20 MB 2025-02-15 04:04:46,686 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:04:46,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:46,687 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:04:46,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:46,688 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:04:46,693 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:04:46,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:04:46,694 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:04:46,694 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:05:50,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:50,335 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:05:50,340 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:05:50,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:50,344 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:05:50,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:50,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:05:53,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:05:53,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:05:53,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.03 seconds 2025-02-15 04:05:53,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:53,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14327.50 MB 2025-02-15 04:05:53,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15017.59 MB 2025-02-15 04:05:53,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-15 04:05:53,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53951.33 MB 2025-02-15 04:05:53,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-15 04:05:53,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33946.60 MB 2025-02-15 04:05:53,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24026.17 MB 2025-02-15 04:05:53,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:05:53,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:05:53,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:05:53,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:53,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15017.59 MB 2025-02-15 04:05:53,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15323.85 MB 2025-02-15 04:05:53,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.26 MB 2025-02-15 04:05:53,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-15 04:05:53,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-15 04:05:53,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:05:53,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17700.45 MB 2025-02-15 04:05:54,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:05:54,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:05:54,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.91 seconds 2025-02-15 04:05:54,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15323.85 MB 2025-02-15 04:05:54,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15577.33 MB 2025-02-15 04:05:54,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 253.48 MB 2025-02-15 04:05:54,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-15 04:05:54,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-15 04:05:54,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 04:05:54,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19515.64 MB 2025-02-15 04:05:54,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:05:54,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:05:54,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:05:54,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15577.26 MB 2025-02-15 04:05:54,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16479.29 MB 2025-02-15 04:05:54,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.03 MB 2025-02-15 04:05:54,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 04:05:54,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-15 04:05:54,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:05:54,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17156.12 MB 2025-02-15 04:05:54,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:05:54,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:05:54,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:05:54,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16479.29 MB 2025-02-15 04:05:54,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17549.81 MB 2025-02-15 04:05:54,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1070.52 MB 2025-02-15 04:05:54,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 04:05:54,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21787.31 MB 2025-02-15 04:05:54,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2254.44 MB 2025-02-15 04:05:54,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20198.22 MB 2025-02-15 04:05:54,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:05:54,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:05:54,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:05:54,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15577.26 MB 2025-02-15 04:05:54,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17549.81 MB 2025-02-15 04:05:54,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1972.55 MB 2025-02-15 04:05:54,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 04:05:54,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21787.31 MB 2025-02-15 04:05:54,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2254.44 MB 2025-02-15 04:05:54,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20198.22 MB 2025-02-15 04:05:54,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:05:54,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:05:54,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:05:54,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18282.08 MB 2025-02-15 04:05:54,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18648.32 MB 2025-02-15 04:05:54,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 366.24 MB 2025-02-15 04:05:54,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21787.31 MB 2025-02-15 04:05:54,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21982.35 MB 2025-02-15 04:05:54,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-15 04:05:54,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18988.86 MB 2025-02-15 04:05:54,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:05:54,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:05:54,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:05:54,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18845.48 MB 2025-02-15 04:05:54,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19070.90 MB 2025-02-15 04:05:54,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.42 MB 2025-02-15 04:05:54,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21982.35 MB 2025-02-15 04:05:54,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21982.35 MB 2025-02-15 04:05:54,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:05:54,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.86 MB 2025-02-15 04:05:54,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:05:54,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:05:54,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.15 seconds 2025-02-15 04:05:54,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13648.10 MB 2025-02-15 04:05:54,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19271.88 MB 2025-02-15 04:05:54,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.77 MB 2025-02-15 04:05:54,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53951.33 MB 2025-02-15 04:05:54,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21982.35 MB 2025-02-15 04:05:54,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31968.99 MB 2025-02-15 04:05:54,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19271.88 MB 2025-02-15 04:05:54,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:05:54,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:05:54,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:05:54,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19271.88 MB 2025-02-15 04:05:54,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17664.63 MB 2025-02-15 04:05:54,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1607.24 MB 2025-02-15 04:05:54,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21982.35 MB 2025-02-15 04:05:54,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21982.35 MB 2025-02-15 04:05:54,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:05:54,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19271.88 MB 2025-02-15 04:05:54,784 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 04:05:54,784 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:05:54,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:05:54,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:05:54,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:05:54,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:05:54,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17664.63 MB 2025-02-15 04:05:54,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26099.48 MB 2025-02-15 04:05:54,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 04:05:54,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21982.35 MB 2025-02-15 04:05:54,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32466.01 MB 2025-02-15 04:05:54,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-15 04:05:54,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26099.48 MB 2025-02-15 04:05:54,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 04:05:54,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:54,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:05:54,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:54,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:05:54,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:05:54,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:05:54,960 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:05:54,960 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:06:21,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:21,518 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:06:21,523 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:06:21,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:21,527 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1312, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:06:21,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:21,528 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1312, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:06:41,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:06:41,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:06:41,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.17 seconds 2025-02-15 04:06:41,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:41,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22110.93 MB 2025-02-15 04:06:41,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26754.03 MB 2025-02-15 04:06:41,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4643.09 MB 2025-02-15 04:06:41,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45044.73 MB 2025-02-15 04:06:41,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37891.34 MB 2025-02-15 04:06:41,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7153.39 MB 2025-02-15 04:06:41,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35659.17 MB 2025-02-15 04:06:41,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:06:41,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:06:41,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:06:41,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:41,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26754.03 MB 2025-02-15 04:06:41,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22598.53 MB 2025-02-15 04:06:41,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4155.50 MB 2025-02-15 04:06:41,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37891.34 MB 2025-02-15 04:06:41,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46672.12 MB 2025-02-15 04:06:41,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8780.78 MB 2025-02-15 04:06:41,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39957.21 MB 2025-02-15 04:06:43,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:06:43,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:06:43,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:06:43,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:43,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22598.53 MB 2025-02-15 04:06:43,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23129.37 MB 2025-02-15 04:06:43,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:06:43,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46672.12 MB 2025-02-15 04:06:43,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 04:06:43,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17616.08 MB 2025-02-15 04:06:43,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27107.92 MB 2025-02-15 04:06:43,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:06:43,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:06:43,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:06:43,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:43,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23129.37 MB 2025-02-15 04:06:43,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25018.90 MB 2025-02-15 04:06:43,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:06:43,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 04:06:43,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 04:06:43,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:06:43,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26436.33 MB 2025-02-15 04:06:43,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:06:43,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:06:43,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:06:43,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:43,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25018.90 MB 2025-02-15 04:06:43,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27260.76 MB 2025-02-15 04:06:43,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:06:43,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 04:06:43,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 04:06:43,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:06:43,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32805.04 MB 2025-02-15 04:06:43,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:06:43,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:06:43,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:06:43,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:43,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23129.37 MB 2025-02-15 04:06:43,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27260.76 MB 2025-02-15 04:06:43,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:06:43,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 04:06:43,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 04:06:43,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:06:43,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32805.04 MB 2025-02-15 04:06:44,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:06:44,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:06:44,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:06:44,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:44,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28794.30 MB 2025-02-15 04:06:44,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29561.30 MB 2025-02-15 04:06:44,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:06:44,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34718.35 MB 2025-02-15 04:06:44,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:06:44,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:06:44,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30269.09 MB 2025-02-15 04:06:44,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:06:44,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:06:44,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:06:44,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:44,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29974.19 MB 2025-02-15 04:06:44,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30202.42 MB 2025-02-15 04:06:44,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 04:06:44,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:06:44,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:06:44,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:06:44,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30411.06 MB 2025-02-15 04:06:44,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:06:44,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:06:44,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.65 seconds 2025-02-15 04:06:44,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:44,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17539.82 MB 2025-02-15 04:06:44,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30402.56 MB 2025-02-15 04:06:44,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12862.74 MB 2025-02-15 04:06:44,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45044.73 MB 2025-02-15 04:06:44,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:06:44,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9909.04 MB 2025-02-15 04:06:44,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30411.06 MB 2025-02-15 04:06:44,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:06:44,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:06:44,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 04:06:44,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:44,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30402.56 MB 2025-02-15 04:06:44,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22530.45 MB 2025-02-15 04:06:44,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7872.11 MB 2025-02-15 04:06:44,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:06:44,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:06:44,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:06:44,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32903.26 MB 2025-02-15 04:06:44,491 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 04:06:44,491 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:06:44,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:06:44,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:06:44,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:06:44,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:06:44,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22530.45 MB 2025-02-15 04:06:44,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30930.35 MB 2025-02-15 04:06:44,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-15 04:06:44,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:06:44,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39311.11 MB 2025-02-15 04:06:44,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 04:06:44,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30930.35 MB 2025-02-15 04:06:44,762 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 04:06:44,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:44,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:06:44,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:44,767 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:06:44,775 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:06:44,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:44,777 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:06:44,777 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:06:55,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:55,279 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:06:55,284 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:06:55,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:55,287 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 582, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:06:55,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:06:55,288 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 582, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:07:04,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:07:04,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:07:04,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.05 seconds 2025-02-15 04:07:04,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:04,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17024.18 MB 2025-02-15 04:07:04,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19083.84 MB 2025-02-15 04:07:04,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2059.67 MB 2025-02-15 04:07:04,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47661.97 MB 2025-02-15 04:07:04,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24861.74 MB 2025-02-15 04:07:04,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22800.24 MB 2025-02-15 04:07:04,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28080.99 MB 2025-02-15 04:07:04,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:07:04,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:07:04,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 04:07:04,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:04,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19083.84 MB 2025-02-15 04:07:04,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18803.49 MB 2025-02-15 04:07:04,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -280.35 MB 2025-02-15 04:07:04,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24861.74 MB 2025-02-15 04:07:04,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29116.86 MB 2025-02-15 04:07:04,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4255.12 MB 2025-02-15 04:07:04,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27101.06 MB 2025-02-15 04:07:06,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:07:06,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:07:06,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 04:07:06,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18803.49 MB 2025-02-15 04:07:06,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19334.33 MB 2025-02-15 04:07:06,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:07:06,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29116.86 MB 2025-02-15 04:07:06,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26277.31 MB 2025-02-15 04:07:06,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2839.54 MB 2025-02-15 04:07:06,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23312.88 MB 2025-02-15 04:07:06,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:07:06,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:07:06,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:07:06,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19334.33 MB 2025-02-15 04:07:06,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21223.86 MB 2025-02-15 04:07:06,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:07:06,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26277.31 MB 2025-02-15 04:07:06,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26277.31 MB 2025-02-15 04:07:06,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:07:06,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22641.29 MB 2025-02-15 04:07:06,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:07:06,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:07:06,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:07:06,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21223.86 MB 2025-02-15 04:07:06,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23465.72 MB 2025-02-15 04:07:06,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:07:06,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26277.31 MB 2025-02-15 04:07:06,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31939.62 MB 2025-02-15 04:07:06,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:07:06,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29010.00 MB 2025-02-15 04:07:06,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:07:06,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:07:06,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:07:06,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19334.33 MB 2025-02-15 04:07:06,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23465.72 MB 2025-02-15 04:07:06,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:07:06,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26277.31 MB 2025-02-15 04:07:06,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31939.62 MB 2025-02-15 04:07:06,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:07:06,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29010.00 MB 2025-02-15 04:07:06,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:07:06,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:07:06,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:07:06,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24999.26 MB 2025-02-15 04:07:06,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25766.26 MB 2025-02-15 04:07:06,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:07:06,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31939.62 MB 2025-02-15 04:07:06,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 04:07:06,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:07:06,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26474.05 MB 2025-02-15 04:07:06,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:07:06,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:07:06,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:07:06,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26179.15 MB 2025-02-15 04:07:06,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26406.41 MB 2025-02-15 04:07:06,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.25 MB 2025-02-15 04:07:06,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 04:07:06,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 04:07:06,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:07:06,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26581.42 MB 2025-02-15 04:07:06,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:07:06,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:07:06,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.41 seconds 2025-02-15 04:07:06,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.44 MB 2025-02-15 04:07:06,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26607.33 MB 2025-02-15 04:07:06,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11610.89 MB 2025-02-15 04:07:06,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47661.97 MB 2025-02-15 04:07:06,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 04:07:06,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15305.02 MB 2025-02-15 04:07:06,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26607.33 MB 2025-02-15 04:07:06,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:07:06,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:07:06,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:07:06,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26607.33 MB 2025-02-15 04:07:06,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19998.54 MB 2025-02-15 04:07:06,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6608.79 MB 2025-02-15 04:07:06,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 04:07:06,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 04:07:06,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:07:06,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29117.15 MB 2025-02-15 04:07:06,988 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 04:07:06,989 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:07:06,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:07:06,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:07:06,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:07:06,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:07:06,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19998.54 MB 2025-02-15 04:07:06,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28431.84 MB 2025-02-15 04:07:06,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 04:07:06,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 04:07:06,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40741.37 MB 2025-02-15 04:07:06,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 04:07:06,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28431.84 MB 2025-02-15 04:07:07,152 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 04:07:07,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:07:07,153 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:07:07,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:07:07,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:07:07,158 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:07:07,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:07:07,160 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:07:07,160 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:08:00,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:00,478 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:08:00,483 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:08:00,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:00,486 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:08:00,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:00,487 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:08:03,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:08:03,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:08:03,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.00 seconds 2025-02-15 04:08:03,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:03,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-15 04:08:03,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.58 MB 2025-02-15 04:08:03,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 683.02 MB 2025-02-15 04:08:03,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49125.79 MB 2025-02-15 04:08:03,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25807.55 MB 2025-02-15 04:08:03,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23318.23 MB 2025-02-15 04:08:03,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24011.42 MB 2025-02-15 04:08:03,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:08:03,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:08:03,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:08:03,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:03,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.58 MB 2025-02-15 04:08:03,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15327.50 MB 2025-02-15 04:08:03,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 330.92 MB 2025-02-15 04:08:03,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25807.55 MB 2025-02-15 04:08:03,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25807.55 MB 2025-02-15 04:08:03,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:03,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17707.53 MB 2025-02-15 04:08:04,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:08:04,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:08:04,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.92 seconds 2025-02-15 04:08:04,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15327.50 MB 2025-02-15 04:08:04,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15583.63 MB 2025-02-15 04:08:04,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 04:08:04,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25807.55 MB 2025-02-15 04:08:04,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25335.69 MB 2025-02-15 04:08:04,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 04:08:04,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19582.08 MB 2025-02-15 04:08:04,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:08:04,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:08:04,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:08:04,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-15 04:08:04,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16495.04 MB 2025-02-15 04:08:04,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 04:08:04,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25335.69 MB 2025-02-15 04:08:04,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25335.69 MB 2025-02-15 04:08:04,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:04,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17178.96 MB 2025-02-15 04:08:04,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:08:04,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:08:04,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:08:04,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16495.04 MB 2025-02-15 04:08:04,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17576.77 MB 2025-02-15 04:08:04,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 04:08:04,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25335.69 MB 2025-02-15 04:08:04,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25335.69 MB 2025-02-15 04:08:04,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:04,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20251.85 MB 2025-02-15 04:08:04,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:08:04,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:08:04,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:08:04,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-15 04:08:04,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17576.77 MB 2025-02-15 04:08:04,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 04:08:04,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25335.69 MB 2025-02-15 04:08:04,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25335.69 MB 2025-02-15 04:08:04,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:04,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20251.85 MB 2025-02-15 04:08:04,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:08:04,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:08:04,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:08:04,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18316.70 MB 2025-02-15 04:08:04,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18686.78 MB 2025-02-15 04:08:04,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 04:08:04,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25335.69 MB 2025-02-15 04:08:04,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-15 04:08:04,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-15 04:08:04,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19030.72 MB 2025-02-15 04:08:04,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:08:04,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:08:04,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:08:04,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18886.01 MB 2025-02-15 04:08:04,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19116.38 MB 2025-02-15 04:08:04,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.37 MB 2025-02-15 04:08:04,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25534.92 MB 2025-02-15 04:08:04,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-15 04:08:04,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:04,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19162.49 MB 2025-02-15 04:08:04,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:08:04,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:08:04,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.14 seconds 2025-02-15 04:08:04,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13641.13 MB 2025-02-15 04:08:04,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19317.45 MB 2025-02-15 04:08:04,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.32 MB 2025-02-15 04:08:04,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49125.79 MB 2025-02-15 04:08:04,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-15 04:08:04,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23590.86 MB 2025-02-15 04:08:04,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19317.45 MB 2025-02-15 04:08:04,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:08:04,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:08:04,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:08:04,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19317.45 MB 2025-02-15 04:08:04,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17668.63 MB 2025-02-15 04:08:04,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1648.82 MB 2025-02-15 04:08:04,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25534.92 MB 2025-02-15 04:08:04,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-15 04:08:04,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:08:04,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19317.45 MB 2025-02-15 04:08:04,917 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:08:04,918 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2,'] 2025-02-15 04:08:04,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:08:04,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:08:04,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:08:04,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:08:04,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.63 MB 2025-02-15 04:08:04,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26107.65 MB 2025-02-15 04:08:04,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:08:04,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25534.92 MB 2025-02-15 04:08:04,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33925.63 MB 2025-02-15 04:08:04,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:08:04,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26107.65 MB 2025-02-15 04:08:05,084 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:08:05,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:05,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:08:05,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:05,086 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:08:05,091 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:08:05,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:08:05,092 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:08:05,092 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2,'] 2025-02-15 04:09:11,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:11,676 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:09:11,681 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:09:11,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:11,685 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1270, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:09:11,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:11,686 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1270, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:09:31,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:09:31,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:09:31,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.44 seconds 2025-02-15 04:09:31,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:31,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21818.27 MB 2025-02-15 04:09:31,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26312.73 MB 2025-02-15 04:09:31,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4494.46 MB 2025-02-15 04:09:31,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46510.64 MB 2025-02-15 04:09:31,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37744.54 MB 2025-02-15 04:09:31,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8766.10 MB 2025-02-15 04:09:31,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35140.01 MB 2025-02-15 04:09:31,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:09:31,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:09:31,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:09:31,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:31,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26312.73 MB 2025-02-15 04:09:31,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22380.18 MB 2025-02-15 04:09:31,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3932.54 MB 2025-02-15 04:09:31,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37744.54 MB 2025-02-15 04:09:31,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46556.77 MB 2025-02-15 04:09:31,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8812.23 MB 2025-02-15 04:09:31,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39558.22 MB 2025-02-15 04:09:33,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:09:33,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:09:33,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 04:09:33,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22380.18 MB 2025-02-15 04:09:33,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22911.03 MB 2025-02-15 04:09:33,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:09:33,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46556.77 MB 2025-02-15 04:09:33,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29053.94 MB 2025-02-15 04:09:33,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17502.83 MB 2025-02-15 04:09:33,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26889.57 MB 2025-02-15 04:09:33,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:09:33,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:09:33,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:09:33,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22911.03 MB 2025-02-15 04:09:33,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24800.56 MB 2025-02-15 04:09:33,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:09:33,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-15 04:09:33,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 04:09:33,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 04:09:33,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26217.99 MB 2025-02-15 04:09:33,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:09:33,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:09:33,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:09:33,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24800.56 MB 2025-02-15 04:09:33,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27042.42 MB 2025-02-15 04:09:33,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:09:33,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 04:09:33,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 04:09:33,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:09:33,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32586.70 MB 2025-02-15 04:09:33,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:09:33,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:09:33,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:09:33,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22911.03 MB 2025-02-15 04:09:33,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27042.42 MB 2025-02-15 04:09:33,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:09:33,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-15 04:09:33,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 04:09:33,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 04:09:33,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32586.70 MB 2025-02-15 04:09:33,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:09:33,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:09:33,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:09:33,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28575.96 MB 2025-02-15 04:09:33,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29342.96 MB 2025-02-15 04:09:33,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:09:33,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34718.35 MB 2025-02-15 04:09:33,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:09:33,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:09:33,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30050.75 MB 2025-02-15 04:09:33,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:09:33,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:09:33,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:09:33,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29755.85 MB 2025-02-15 04:09:33,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29984.61 MB 2025-02-15 04:09:33,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.76 MB 2025-02-15 04:09:33,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:09:33,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:09:33,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:09:33,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30217.94 MB 2025-02-15 04:09:33,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:09:33,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:09:33,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.84 seconds 2025-02-15 04:09:33,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17393.49 MB 2025-02-15 04:09:33,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30185.29 MB 2025-02-15 04:09:33,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12791.80 MB 2025-02-15 04:09:33,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46510.64 MB 2025-02-15 04:09:33,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:09:33,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11374.95 MB 2025-02-15 04:09:33,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30217.94 MB 2025-02-15 04:09:33,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:09:33,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:09:33,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:09:33,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30185.29 MB 2025-02-15 04:09:33,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22391.95 MB 2025-02-15 04:09:33,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7793.34 MB 2025-02-15 04:09:33,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:09:33,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 04:09:33,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:09:33,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32692.04 MB 2025-02-15 04:09:33,815 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 04:09:33,816 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:09:33,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:09:33,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:09:33,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:09:33,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:09:33,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22391.95 MB 2025-02-15 04:09:33,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30814.13 MB 2025-02-15 04:09:33,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.18 MB 2025-02-15 04:09:33,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 04:09:33,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-15 04:09:33,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 04:09:33,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30814.13 MB 2025-02-15 04:09:33,982 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 04:09:33,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:33,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:09:33,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:33,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:09:33,989 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:09:33,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:09:33,991 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:09:33,991 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:10:31,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:31,321 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:10:31,327 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:10:31,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:31,331 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1501, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:10:31,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:31,332 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1501, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:10:54,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:10:54,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:10:54,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.09 seconds 2025-02-15 04:10:54,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:54,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23427.91 MB 2025-02-15 04:10:54,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28740.00 MB 2025-02-15 04:10:54,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5312.09 MB 2025-02-15 04:10:54,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47695.53 MB 2025-02-15 04:10:54,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38541.46 MB 2025-02-15 04:10:54,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9154.07 MB 2025-02-15 04:10:54,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37655.63 MB 2025-02-15 04:10:54,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:10:54,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:10:54,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:10:54,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:54,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28740.00 MB 2025-02-15 04:10:54,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23581.08 MB 2025-02-15 04:10:54,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5158.92 MB 2025-02-15 04:10:54,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38541.46 MB 2025-02-15 04:10:54,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48769.27 MB 2025-02-15 04:10:54,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10227.81 MB 2025-02-15 04:10:54,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44119.82 MB 2025-02-15 04:10:56,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:10:56,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:10:56,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:10:56,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23581.08 MB 2025-02-15 04:10:56,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24111.92 MB 2025-02-15 04:10:56,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:10:56,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48769.27 MB 2025-02-15 04:10:56,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-15 04:10:56,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19727.91 MB 2025-02-15 04:10:56,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28090.47 MB 2025-02-15 04:10:56,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:10:56,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:10:56,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:10:56,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24111.92 MB 2025-02-15 04:10:56,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26001.46 MB 2025-02-15 04:10:56,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:10:56,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 04:10:56,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-15 04:10:56,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:10:56,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27418.89 MB 2025-02-15 04:10:56,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:10:56,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:10:56,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:10:56,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26001.46 MB 2025-02-15 04:10:56,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28243.31 MB 2025-02-15 04:10:56,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:10:56,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 04:10:56,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-15 04:10:56,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 04:10:56,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33787.59 MB 2025-02-15 04:10:56,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:10:56,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:10:56,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:10:56,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24111.92 MB 2025-02-15 04:10:56,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28243.31 MB 2025-02-15 04:10:56,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:10:56,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 04:10:56,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-15 04:10:56,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 04:10:56,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33787.59 MB 2025-02-15 04:10:56,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:10:56,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:10:56,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:10:56,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29776.86 MB 2025-02-15 04:10:56,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30543.86 MB 2025-02-15 04:10:56,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:10:56,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36356.23 MB 2025-02-15 04:10:56,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36773.56 MB 2025-02-15 04:10:56,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:10:56,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31251.65 MB 2025-02-15 04:10:56,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:10:56,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:10:56,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:10:56,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30956.75 MB 2025-02-15 04:10:56,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31185.51 MB 2025-02-15 04:10:56,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.76 MB 2025-02-15 04:10:56,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36773.56 MB 2025-02-15 04:10:56,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36773.56 MB 2025-02-15 04:10:56,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:10:56,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31402.36 MB 2025-02-15 04:10:56,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:10:56,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:10:56,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.55 seconds 2025-02-15 04:10:56,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:56,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18198.31 MB 2025-02-15 04:10:56,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31386.36 MB 2025-02-15 04:10:56,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13188.05 MB 2025-02-15 04:10:56,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47695.53 MB 2025-02-15 04:10:56,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36773.56 MB 2025-02-15 04:10:56,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10921.97 MB 2025-02-15 04:10:56,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31402.36 MB 2025-02-15 04:10:57,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:10:57,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:10:57,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:10:57,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:57,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31386.36 MB 2025-02-15 04:10:57,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23197.13 MB 2025-02-15 04:10:57,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8189.23 MB 2025-02-15 04:10:57,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36773.56 MB 2025-02-15 04:10:57,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36773.56 MB 2025-02-15 04:10:57,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:10:57,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33893.42 MB 2025-02-15 04:10:57,173 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 04:10:57,173 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:10:57,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:10:57,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:10:57,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:10:57,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:10:57,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23197.13 MB 2025-02-15 04:10:57,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31620.34 MB 2025-02-15 04:10:57,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 04:10:57,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36773.56 MB 2025-02-15 04:10:57,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45149.59 MB 2025-02-15 04:10:57,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 04:10:57,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31620.34 MB 2025-02-15 04:10:57,341 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 04:10:57,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:57,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:10:57,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:57,343 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:10:57,348 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:10:57,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:10:57,349 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:10:57,349 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:11:16,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:16,589 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:11:16,594 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:11:16,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:16,598 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1434, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:11:16,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:16,599 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1434, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:11:38,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:11:38,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:11:38,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.38 seconds 2025-02-15 04:11:38,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:38,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22961.05 MB 2025-02-15 04:11:38,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28036.16 MB 2025-02-15 04:11:38,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5075.11 MB 2025-02-15 04:11:38,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53525.61 MB 2025-02-15 04:11:38,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38302.38 MB 2025-02-15 04:11:38,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15223.23 MB 2025-02-15 04:11:38,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36962.27 MB 2025-02-15 04:11:39,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:11:39,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:11:39,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:11:39,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:39,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28036.16 MB 2025-02-15 04:11:39,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23232.77 MB 2025-02-15 04:11:39,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4803.39 MB 2025-02-15 04:11:39,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38302.38 MB 2025-02-15 04:11:39,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 04:11:39,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9877.59 MB 2025-02-15 04:11:39,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42888.10 MB 2025-02-15 04:11:41,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:11:41,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:11:41,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:11:41,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23232.77 MB 2025-02-15 04:11:41,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23763.61 MB 2025-02-15 04:11:41,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:11:41,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 04:11:41,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33227.28 MB 2025-02-15 04:11:41,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14952.69 MB 2025-02-15 04:11:41,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27742.16 MB 2025-02-15 04:11:41,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:11:41,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:11:41,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:11:41,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-15 04:11:41,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25653.14 MB 2025-02-15 04:11:41,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:11:41,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33227.28 MB 2025-02-15 04:11:41,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33227.28 MB 2025-02-15 04:11:41,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:11:41,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27070.57 MB 2025-02-15 04:11:41,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:11:41,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:11:41,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:11:41,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25653.14 MB 2025-02-15 04:11:41,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-15 04:11:41,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:11:41,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33227.28 MB 2025-02-15 04:11:41,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 04:11:41,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:11:41,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-15 04:11:41,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:11:41,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:11:41,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:11:41,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-15 04:11:41,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-15 04:11:41,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:11:41,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33227.28 MB 2025-02-15 04:11:41,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 04:11:41,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:11:41,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-15 04:11:41,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:11:41,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:11:41,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:11:41,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29428.54 MB 2025-02-15 04:11:41,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30195.54 MB 2025-02-15 04:11:41,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:11:41,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 04:11:41,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36473.67 MB 2025-02-15 04:11:41,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:11:41,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30903.33 MB 2025-02-15 04:11:41,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:11:41,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:11:41,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:11:41,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30608.43 MB 2025-02-15 04:11:41,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30836.46 MB 2025-02-15 04:11:41,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-15 04:11:41,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36473.67 MB 2025-02-15 04:11:41,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36473.67 MB 2025-02-15 04:11:41,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:11:41,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31072.02 MB 2025-02-15 04:11:41,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:11:41,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:11:41,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.81 seconds 2025-02-15 04:11:41,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17964.88 MB 2025-02-15 04:11:41,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31037.31 MB 2025-02-15 04:11:41,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13072.44 MB 2025-02-15 04:11:41,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53525.61 MB 2025-02-15 04:11:41,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36473.67 MB 2025-02-15 04:11:41,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17051.94 MB 2025-02-15 04:11:41,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31072.02 MB 2025-02-15 04:11:41,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:11:41,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:11:41,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:11:41,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31037.31 MB 2025-02-15 04:11:41,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22952.65 MB 2025-02-15 04:11:41,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8084.66 MB 2025-02-15 04:11:41,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36473.67 MB 2025-02-15 04:11:41,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36473.67 MB 2025-02-15 04:11:41,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:11:41,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33534.85 MB 2025-02-15 04:11:41,700 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 04:11:41,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:11:41,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:11:41,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:11:41,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:11:41,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:11:41,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22952.65 MB 2025-02-15 04:11:41,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31345.08 MB 2025-02-15 04:11:41,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-15 04:11:41,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36473.67 MB 2025-02-15 04:11:41,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44816.14 MB 2025-02-15 04:11:41,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 04:11:41,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31345.08 MB 2025-02-15 04:11:41,870 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 04:11:41,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:41,872 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:11:41,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:41,873 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:11:41,877 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:11:41,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:41,878 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:11:41,878 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:11:55,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:55,448 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:11:55,453 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:11:55,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:55,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 442, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:11:55,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:11:55,457 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 442, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:12:02,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:12:02,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:12:02,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.90 seconds 2025-02-15 04:12:02,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:02,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.63 MB 2025-02-15 04:12:02,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17612.85 MB 2025-02-15 04:12:02,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1564.21 MB 2025-02-15 04:12:02,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53158.61 MB 2025-02-15 04:12:02,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22554.87 MB 2025-02-15 04:12:02,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30603.74 MB 2025-02-15 04:12:02,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26425.97 MB 2025-02-15 04:12:02,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:12:02,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:12:02,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:12:02,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:02,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17612.85 MB 2025-02-15 04:12:02,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18020.53 MB 2025-02-15 04:12:02,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 407.69 MB 2025-02-15 04:12:02,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22554.87 MB 2025-02-15 04:12:02,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25331.50 MB 2025-02-15 04:12:02,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2776.63 MB 2025-02-15 04:12:02,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23120.99 MB 2025-02-15 04:12:04,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:12:04,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:12:04,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 04:12:04,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.53 MB 2025-02-15 04:12:04,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18540.76 MB 2025-02-15 04:12:04,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 520.22 MB 2025-02-15 04:12:04,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25331.50 MB 2025-02-15 04:12:04,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22999.47 MB 2025-02-15 04:12:04,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2332.03 MB 2025-02-15 04:12:04,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22530.96 MB 2025-02-15 04:12:04,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:12:04,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:12:04,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:12:04,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-15 04:12:04,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20392.54 MB 2025-02-15 04:12:04,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1851.79 MB 2025-02-15 04:12:04,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22999.47 MB 2025-02-15 04:12:04,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23924.31 MB 2025-02-15 04:12:04,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 924.84 MB 2025-02-15 04:12:04,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21781.63 MB 2025-02-15 04:12:04,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:12:04,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:12:04,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:12:04,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20392.54 MB 2025-02-15 04:12:04,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-15 04:12:04,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2197.02 MB 2025-02-15 04:12:04,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23924.31 MB 2025-02-15 04:12:04,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30171.73 MB 2025-02-15 04:12:04,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6247.42 MB 2025-02-15 04:12:04,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-15 04:12:04,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:12:04,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:12:04,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:12:04,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-15 04:12:04,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-15 04:12:04,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4048.81 MB 2025-02-15 04:12:04,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22999.47 MB 2025-02-15 04:12:04,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30171.73 MB 2025-02-15 04:12:04,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7172.26 MB 2025-02-15 04:12:04,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-15 04:12:04,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:12:04,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:12:04,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:12:04,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24092.44 MB 2025-02-15 04:12:04,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24846.20 MB 2025-02-15 04:12:04,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 753.76 MB 2025-02-15 04:12:04,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30171.73 MB 2025-02-15 04:12:04,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30578.57 MB 2025-02-15 04:12:04,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 406.85 MB 2025-02-15 04:12:04,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25539.83 MB 2025-02-15 04:12:04,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:12:04,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:12:04,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:12:04,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.83 MB 2025-02-15 04:12:04,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25482.49 MB 2025-02-15 04:12:04,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.67 MB 2025-02-15 04:12:04,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30578.57 MB 2025-02-15 04:12:04,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30578.57 MB 2025-02-15 04:12:04,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:12:04,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25633.87 MB 2025-02-15 04:12:04,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:12:04,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:12:04,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.23 seconds 2025-02-15 04:12:04,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14508.67 MB 2025-02-15 04:12:04,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25683.57 MB 2025-02-15 04:12:04,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11174.90 MB 2025-02-15 04:12:04,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53158.61 MB 2025-02-15 04:12:04,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30578.57 MB 2025-02-15 04:12:04,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22580.04 MB 2025-02-15 04:12:04,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25683.57 MB 2025-02-15 04:12:04,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:12:04,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:12:04,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:12:04,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25683.57 MB 2025-02-15 04:12:04,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19476.88 MB 2025-02-15 04:12:04,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6206.69 MB 2025-02-15 04:12:04,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30578.57 MB 2025-02-15 04:12:04,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30578.57 MB 2025-02-15 04:12:04,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:12:04,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28295.70 MB 2025-02-15 04:12:04,982 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:12:04,982 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:12:04,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:12:04,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:12:04,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:12:04,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:12:04,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19476.88 MB 2025-02-15 04:12:04,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27915.90 MB 2025-02-15 04:12:04,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:12:04,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30578.57 MB 2025-02-15 04:12:04,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38969.28 MB 2025-02-15 04:12:04,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:12:04,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27915.90 MB 2025-02-15 04:12:05,151 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:12:05,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:12:05,153 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:12:05,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:12:05,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:12:05,158 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:12:05,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:12:05,159 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:12:05,160 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:13:07,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:07,640 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:13:07,648 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:13:07,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:07,655 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 212, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:13:07,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:07,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 212, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:13:11,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:13:11,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:13:11,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.34 seconds 2025-02-15 04:13:11,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:11,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.96 MB 2025-02-15 04:13:11,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.21 MB 2025-02-15 04:13:11,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.26 MB 2025-02-15 04:13:11,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51554.29 MB 2025-02-15 04:13:11,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17871.93 MB 2025-02-15 04:13:11,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33682.36 MB 2025-02-15 04:13:11,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24143.82 MB 2025-02-15 04:13:11,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:13:11,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:13:11,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:13:11,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:11,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.21 MB 2025-02-15 04:13:11,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15434.11 MB 2025-02-15 04:13:11,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.90 MB 2025-02-15 04:13:11,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17871.93 MB 2025-02-15 04:13:11,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19581.11 MB 2025-02-15 04:13:11,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1709.18 MB 2025-02-15 04:13:11,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17922.02 MB 2025-02-15 04:13:11,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:13:11,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:13:11,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 04:13:11,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:11,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15434.11 MB 2025-02-15 04:13:11,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15691.57 MB 2025-02-15 04:13:11,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 257.46 MB 2025-02-15 04:13:11,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19581.11 MB 2025-02-15 04:13:11,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18429.77 MB 2025-02-15 04:13:11,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1151.34 MB 2025-02-15 04:13:11,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19689.73 MB 2025-02-15 04:13:11,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:13:11,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:13:11,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:13:11,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:11,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15691.50 MB 2025-02-15 04:13:11,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16607.70 MB 2025-02-15 04:13:11,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 916.20 MB 2025-02-15 04:13:11,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18429.77 MB 2025-02-15 04:13:11,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18889.05 MB 2025-02-15 04:13:11,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 459.28 MB 2025-02-15 04:13:11,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17295.16 MB 2025-02-15 04:13:12,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:13:12,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:13:12,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:13:12,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16607.70 MB 2025-02-15 04:13:12,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17695.04 MB 2025-02-15 04:13:12,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1087.33 MB 2025-02-15 04:13:12,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 04:13:12,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21646.80 MB 2025-02-15 04:13:12,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2757.75 MB 2025-02-15 04:13:12,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20383.98 MB 2025-02-15 04:13:12,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:13:12,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:13:12,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:13:12,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15691.50 MB 2025-02-15 04:13:12,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17695.04 MB 2025-02-15 04:13:12,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2003.54 MB 2025-02-15 04:13:12,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18429.77 MB 2025-02-15 04:13:12,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21646.80 MB 2025-02-15 04:13:12,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3217.03 MB 2025-02-15 04:13:12,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20383.98 MB 2025-02-15 04:13:12,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:13:12,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:13:12,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:13:12,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18438.80 MB 2025-02-15 04:13:12,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18810.80 MB 2025-02-15 04:13:12,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 372.00 MB 2025-02-15 04:13:12,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21646.80 MB 2025-02-15 04:13:12,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21843.94 MB 2025-02-15 04:13:12,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-15 04:13:12,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19158.35 MB 2025-02-15 04:13:12,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:13:12,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:13:12,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:13:12,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19011.06 MB 2025-02-15 04:13:12,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19215.59 MB 2025-02-15 04:13:12,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.53 MB 2025-02-15 04:13:12,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21843.94 MB 2025-02-15 04:13:12,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21848.13 MB 2025-02-15 04:13:12,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 04:13:12,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19263.75 MB 2025-02-15 04:13:12,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:13:12,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:13:12,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.50 seconds 2025-02-15 04:13:12,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-15 04:13:12,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19416.66 MB 2025-02-15 04:13:12,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5709.33 MB 2025-02-15 04:13:12,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51554.29 MB 2025-02-15 04:13:12,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21848.13 MB 2025-02-15 04:13:12,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29706.16 MB 2025-02-15 04:13:12,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19416.66 MB 2025-02-15 04:13:12,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:13:12,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:13:12,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:13:12,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14724.99 MB 2025-02-15 04:13:12,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.02 MB 2025-02-15 04:13:12,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:13:12,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21848.13 MB 2025-02-15 04:13:12,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21848.13 MB 2025-02-15 04:13:12,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:13:12,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18040.39 MB 2025-02-15 04:13:12,449 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:13:12,449 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:13:12,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:13:12,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:13:12,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:13:12,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:12,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17739.02 MB 2025-02-15 04:13:12,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26178.05 MB 2025-02-15 04:13:12,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:13:12,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21848.13 MB 2025-02-15 04:13:12,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32338.08 MB 2025-02-15 04:13:12,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:13:12,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26178.05 MB 2025-02-15 04:13:12,615 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:13:12,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:12,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:13:12,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:12,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:13:12,622 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:13:12,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:12,623 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:13:12,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:13:28,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:28,316 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:13:28,321 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:13:28,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:28,325 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1377, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:13:28,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:28,326 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1377, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:13:49,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:13:49,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:13:49,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.21 seconds 2025-02-15 04:13:49,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:49,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22563.86 MB 2025-02-15 04:13:49,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27437.64 MB 2025-02-15 04:13:49,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4873.78 MB 2025-02-15 04:13:49,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44923.09 MB 2025-02-15 04:13:49,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38134.61 MB 2025-02-15 04:13:49,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6788.48 MB 2025-02-15 04:13:49,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36338.59 MB 2025-02-15 04:13:49,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:13:49,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:13:49,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:13:49,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:49,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27437.64 MB 2025-02-15 04:13:49,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22936.44 MB 2025-02-15 04:13:49,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4501.20 MB 2025-02-15 04:13:49,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38134.61 MB 2025-02-15 04:13:49,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47758.44 MB 2025-02-15 04:13:49,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9623.83 MB 2025-02-15 04:13:49,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41908.72 MB 2025-02-15 04:13:51,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:13:51,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:13:51,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:13:51,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22936.44 MB 2025-02-15 04:13:51,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23467.29 MB 2025-02-15 04:13:51,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:13:51,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47758.44 MB 2025-02-15 04:13:51,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29066.53 MB 2025-02-15 04:13:51,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18691.92 MB 2025-02-15 04:13:51,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27445.83 MB 2025-02-15 04:13:51,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:13:51,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:13:51,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:13:51,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23467.29 MB 2025-02-15 04:13:51,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25356.82 MB 2025-02-15 04:13:51,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:13:51,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 04:13:51,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30010.25 MB 2025-02-15 04:13:51,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:13:51,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26774.25 MB 2025-02-15 04:13:51,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:13:51,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:13:51,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:13:51,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25356.82 MB 2025-02-15 04:13:51,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27598.68 MB 2025-02-15 04:13:51,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:13:51,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30010.25 MB 2025-02-15 04:13:51,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 04:13:51,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:13:51,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33142.96 MB 2025-02-15 04:13:51,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:13:51,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:13:51,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:13:51,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23467.29 MB 2025-02-15 04:13:51,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27598.68 MB 2025-02-15 04:13:51,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:13:51,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 04:13:51,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 04:13:51,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:13:51,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33142.96 MB 2025-02-15 04:13:51,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:13:51,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:13:51,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:13:51,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29132.22 MB 2025-02-15 04:13:51,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29899.22 MB 2025-02-15 04:13:51,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:13:51,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35672.56 MB 2025-02-15 04:13:51,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:13:51,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:13:51,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30607.01 MB 2025-02-15 04:13:51,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:13:51,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:13:51,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:13:51,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.11 MB 2025-02-15 04:13:51,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30540.06 MB 2025-02-15 04:13:51,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-15 04:13:51,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:13:51,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:13:51,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:13:51,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30769.13 MB 2025-02-15 04:13:51,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:13:51,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:13:51,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.64 seconds 2025-02-15 04:13:51,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:51,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17766.28 MB 2025-02-15 04:13:51,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30740.91 MB 2025-02-15 04:13:51,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12974.63 MB 2025-02-15 04:13:51,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44923.09 MB 2025-02-15 04:13:51,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:13:51,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8837.40 MB 2025-02-15 04:13:51,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30769.13 MB 2025-02-15 04:13:52,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:13:52,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:13:52,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:13:52,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:52,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30740.91 MB 2025-02-15 04:13:52,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22752.99 MB 2025-02-15 04:13:52,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7987.92 MB 2025-02-15 04:13:52,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:13:52,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 04:13:52,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:13:52,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33237.53 MB 2025-02-15 04:13:52,252 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 04:13:52,253 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:13:52,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:13:52,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:13:52,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:13:52,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:13:52,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22752.99 MB 2025-02-15 04:13:52,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31141.41 MB 2025-02-15 04:13:52,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 04:13:52,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 04:13:52,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44426.07 MB 2025-02-15 04:13:52,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-15 04:13:52,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31141.41 MB 2025-02-15 04:13:52,417 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 04:13:52,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:52,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:13:52,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:52,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:13:52,424 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:13:52,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:13:52,425 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:13:52,425 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:14:03,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:03,779 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:14:03,784 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:14:03,787 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:03,788 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 300, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:14:03,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:03,789 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 300, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:14:08,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:14:08,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:14:08,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.68 seconds 2025-02-15 04:14:08,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15059.15 MB 2025-02-15 04:14:08,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16120.84 MB 2025-02-15 04:14:08,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1061.68 MB 2025-02-15 04:14:08,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56935.58 MB 2025-02-15 04:14:08,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-15 04:14:08,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35985.03 MB 2025-02-15 04:14:08,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24983.51 MB 2025-02-15 04:14:08,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:14:08,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:14:08,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:14:08,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16120.84 MB 2025-02-15 04:14:08,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14661.75 MB 2025-02-15 04:14:08,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1459.09 MB 2025-02-15 04:14:08,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-15 04:14:08,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-15 04:14:08,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16387.78 MB 2025-02-15 04:14:08,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:14:08,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:14:08,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:14:08,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14661.75 MB 2025-02-15 04:14:08,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14686.97 MB 2025-02-15 04:14:08,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.21 MB 2025-02-15 04:14:08,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-15 04:14:08,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 04:14:08,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2581.59 MB 2025-02-15 04:14:08,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15875.00 MB 2025-02-15 04:14:08,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:14:08,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:14:08,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:14:08,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14686.90 MB 2025-02-15 04:14:08,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14776.63 MB 2025-02-15 04:14:08,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 89.73 MB 2025-02-15 04:14:08,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 04:14:08,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 04:14:08,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14843.97 MB 2025-02-15 04:14:08,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:14:08,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:14:08,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:14:08,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14776.63 MB 2025-02-15 04:14:08,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14883.77 MB 2025-02-15 04:14:08,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 107.14 MB 2025-02-15 04:14:08,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 04:14:08,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 04:14:08,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15147.76 MB 2025-02-15 04:14:08,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:14:08,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:14:08,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:14:08,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14686.90 MB 2025-02-15 04:14:08,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14883.77 MB 2025-02-15 04:14:08,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.87 MB 2025-02-15 04:14:08,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 04:14:08,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 04:14:08,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15147.76 MB 2025-02-15 04:14:08,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:14:08,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:14:08,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:14:08,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14957.31 MB 2025-02-15 04:14:08,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14993.74 MB 2025-02-15 04:14:08,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 36.43 MB 2025-02-15 04:14:08,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 04:14:08,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18381.54 MB 2025-02-15 04:14:08,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12.58 MB 2025-02-15 04:14:08,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15041.99 MB 2025-02-15 04:14:08,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:14:08,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:14:08,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:14:08,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15013.36 MB 2025-02-15 04:14:08,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15037.95 MB 2025-02-15 04:14:08,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 24.59 MB 2025-02-15 04:14:08,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18381.54 MB 2025-02-15 04:14:08,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18381.54 MB 2025-02-15 04:14:08,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15037.95 MB 2025-02-15 04:14:08,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:14:08,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:14:08,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.84 seconds 2025-02-15 04:14:08,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14013.93 MB 2025-02-15 04:14:08,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15083.02 MB 2025-02-15 04:14:08,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1069.09 MB 2025-02-15 04:14:08,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56935.58 MB 2025-02-15 04:14:08,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18381.54 MB 2025-02-15 04:14:08,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38554.04 MB 2025-02-15 04:14:08,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15083.02 MB 2025-02-15 04:14:08,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:14:08,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:14:08,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:14:08,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15083.02 MB 2025-02-15 04:14:08,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15758.75 MB 2025-02-15 04:14:08,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.72 MB 2025-02-15 04:14:08,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18381.54 MB 2025-02-15 04:14:08,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18387.83 MB 2025-02-15 04:14:08,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6.29 MB 2025-02-15 04:14:08,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15826.31 MB 2025-02-15 04:14:08,703 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1819, cut from 1821 2025-02-15 04:14:08,703 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:14:08,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:14:08,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:14:08,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:14:08,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:14:08,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14804.59 MB 2025-02-15 04:14:08,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16696.09 MB 2025-02-15 04:14:08,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1891.50 MB 2025-02-15 04:14:08,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18387.83 MB 2025-02-15 04:14:08,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18387.83 MB 2025-02-15 04:14:08,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:14:08,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16696.09 MB 2025-02-15 04:14:08,741 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1611] 2025-02-15 04:14:08,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:08,742 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:14:08,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:08,743 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:14:08,748 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:14:08,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:14:08,749 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:14:08,749 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:15:25,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:25,291 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:15:25,299 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:15:25,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:25,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:15:25,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:25,307 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:15:28,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:15:28,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:15:28,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.19 seconds 2025-02-15 04:15:28,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:28,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-15 04:15:28,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-15 04:15:28,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-15 04:15:28,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20271.07 MB 2025-02-15 04:15:28,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-15 04:15:28,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -268.44 MB 2025-02-15 04:15:28,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.07 MB 2025-02-15 04:15:28,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:15:28,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:15:28,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:15:28,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:28,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-15 04:15:28,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15384.68 MB 2025-02-15 04:15:28,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.53 MB 2025-02-15 04:15:28,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-15 04:15:28,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-15 04:15:28,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:15:28,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17823.11 MB 2025-02-15 04:15:29,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:15:29,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:15:29,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 04:15:29,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15384.68 MB 2025-02-15 04:15:29,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15640.81 MB 2025-02-15 04:15:29,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 04:15:29,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-15 04:15:29,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 04:15:29,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 04:15:29,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19640.31 MB 2025-02-15 04:15:29,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:15:29,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:15:29,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:15:29,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 04:15:29,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.23 MB 2025-02-15 04:15:29,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 04:15:29,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 04:15:29,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18821.94 MB 2025-02-15 04:15:29,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-15 04:15:29,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.14 MB 2025-02-15 04:15:29,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:15:29,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:15:29,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:15:29,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.23 MB 2025-02-15 04:15:29,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 04:15:29,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 04:15:29,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18821.94 MB 2025-02-15 04:15:29,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21565.01 MB 2025-02-15 04:15:29,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-15 04:15:29,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 04:15:29,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:15:29,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:15:29,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:15:29,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 04:15:29,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 04:15:29,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 04:15:29,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 04:15:29,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21565.01 MB 2025-02-15 04:15:29,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3200.25 MB 2025-02-15 04:15:29,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 04:15:29,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:15:29,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:15:29,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:15:29,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.89 MB 2025-02-15 04:15:29,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18743.97 MB 2025-02-15 04:15:29,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 04:15:29,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21565.01 MB 2025-02-15 04:15:29,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21764.24 MB 2025-02-15 04:15:29,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-15 04:15:29,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19090.45 MB 2025-02-15 04:15:29,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:15:29,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:15:29,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:15:29,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18943.20 MB 2025-02-15 04:15:29,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19172.03 MB 2025-02-15 04:15:29,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.83 MB 2025-02-15 04:15:29,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21764.24 MB 2025-02-15 04:15:29,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21764.24 MB 2025-02-15 04:15:29,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:15:29,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19205.74 MB 2025-02-15 04:15:29,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:15:29,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:15:29,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.34 seconds 2025-02-15 04:15:29,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-15 04:15:29,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19373.10 MB 2025-02-15 04:15:29,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5693.64 MB 2025-02-15 04:15:29,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20271.07 MB 2025-02-15 04:15:29,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21766.34 MB 2025-02-15 04:15:29,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1495.27 MB 2025-02-15 04:15:29,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.10 MB 2025-02-15 04:15:29,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:15:29,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:15:29,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:15:29,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19373.10 MB 2025-02-15 04:15:29,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17706.69 MB 2025-02-15 04:15:29,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1666.41 MB 2025-02-15 04:15:29,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21766.34 MB 2025-02-15 04:15:29,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21766.34 MB 2025-02-15 04:15:29,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:15:29,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.10 MB 2025-02-15 04:15:29,938 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:15:29,938 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:15:29,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:15:29,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:15:29,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:15:29,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:15:29,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17706.69 MB 2025-02-15 04:15:29,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26145.72 MB 2025-02-15 04:15:29,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:15:29,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21766.34 MB 2025-02-15 04:15:29,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32256.29 MB 2025-02-15 04:15:29,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:15:29,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26145.72 MB 2025-02-15 04:15:30,102 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:15:30,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:30,103 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:15:30,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:30,104 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:15:30,108 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:15:30,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:15:30,110 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:15:30,110 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:16:26,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:26,659 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:16:26,664 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:16:26,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:26,669 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1720, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:16:26,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:26,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1720, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:16:53,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:16:53,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:16:53,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.44 seconds 2025-02-15 04:16:53,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:53,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24953.94 MB 2025-02-15 04:16:53,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31041.97 MB 2025-02-15 04:16:53,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6088.03 MB 2025-02-15 04:16:53,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44841.30 MB 2025-02-15 04:16:53,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39342.57 MB 2025-02-15 04:16:53,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5498.73 MB 2025-02-15 04:16:53,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39861.13 MB 2025-02-15 04:16:53,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:16:53,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:16:53,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 04:16:53,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:53,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31041.97 MB 2025-02-15 04:16:53,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24719.59 MB 2025-02-15 04:16:53,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6322.38 MB 2025-02-15 04:16:53,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39342.57 MB 2025-02-15 04:16:53,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52544.14 MB 2025-02-15 04:16:53,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13201.57 MB 2025-02-15 04:16:53,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48768.00 MB 2025-02-15 04:16:55,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:16:55,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:16:55,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:16:55,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24719.59 MB 2025-02-15 04:16:55,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25250.44 MB 2025-02-15 04:16:55,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:16:55,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52544.14 MB 2025-02-15 04:16:55,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-15 04:16:55,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17874.03 MB 2025-02-15 04:16:55,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29228.98 MB 2025-02-15 04:16:55,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:16:55,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:16:55,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:16:55,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-15 04:16:55,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27139.97 MB 2025-02-15 04:16:55,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:16:55,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 04:16:55,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-15 04:16:55,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:16:55,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28557.40 MB 2025-02-15 04:16:55,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:16:55,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:16:55,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:16:55,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27139.97 MB 2025-02-15 04:16:55,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-15 04:16:55,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:16:55,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 04:16:55,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37503.37 MB 2025-02-15 04:16:55,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 04:16:55,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-15 04:16:55,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:16:55,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:16:55,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:16:55,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-15 04:16:55,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-15 04:16:55,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:16:55,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 04:16:55,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37503.37 MB 2025-02-15 04:16:55,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 04:16:55,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-15 04:16:55,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:16:55,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:16:55,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:16:55,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30915.37 MB 2025-02-15 04:16:55,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31682.37 MB 2025-02-15 04:16:55,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:16:55,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37503.37 MB 2025-02-15 04:16:55,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:16:55,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:16:55,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32390.16 MB 2025-02-15 04:16:55,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:16:55,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:16:55,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:16:55,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32095.26 MB 2025-02-15 04:16:55,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32323.58 MB 2025-02-15 04:16:55,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-15 04:16:55,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:16:55,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:16:55,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:16:55,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32531.73 MB 2025-02-15 04:16:55,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:16:55,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:16:55,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.91 seconds 2025-02-15 04:16:55,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18961.32 MB 2025-02-15 04:16:55,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32524.43 MB 2025-02-15 04:16:55,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13563.11 MB 2025-02-15 04:16:55,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44841.30 MB 2025-02-15 04:16:55,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:16:55,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6920.60 MB 2025-02-15 04:16:55,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32531.73 MB 2025-02-15 04:16:55,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:16:55,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:16:55,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:16:55,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32524.43 MB 2025-02-15 04:16:55,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23953.38 MB 2025-02-15 04:16:55,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8571.06 MB 2025-02-15 04:16:55,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:16:55,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:16:55,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:16:55,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35025.65 MB 2025-02-15 04:16:55,867 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 04:16:55,867 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:16:55,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:16:55,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:16:55,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:16:55,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:16:55,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23953.38 MB 2025-02-15 04:16:55,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32357.97 MB 2025-02-15 04:16:55,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.59 MB 2025-02-15 04:16:55,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:16:55,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42098.23 MB 2025-02-15 04:16:55,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 04:16:55,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32357.97 MB 2025-02-15 04:16:56,038 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 04:16:56,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:56,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:16:56,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:56,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:16:56,045 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:16:56,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:16:56,046 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:16:56,046 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:17:51,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:17:51,079 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:17:51,086 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:17:51,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:17:51,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1484, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:17:51,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:17:51,095 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1484, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:18:14,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:18:14,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:18:14,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.93 seconds 2025-02-15 04:18:14,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:14,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23309.46 MB 2025-02-15 04:18:14,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28561.25 MB 2025-02-15 04:18:14,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5251.79 MB 2025-02-15 04:18:14,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50453.28 MB 2025-02-15 04:18:14,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38457.57 MB 2025-02-15 04:18:14,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11995.71 MB 2025-02-15 04:18:14,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37537.17 MB 2025-02-15 04:18:14,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:18:14,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:18:14,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:18:14,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:14,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28561.25 MB 2025-02-15 04:18:14,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23492.70 MB 2025-02-15 04:18:14,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5068.55 MB 2025-02-15 04:18:14,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38457.57 MB 2025-02-15 04:18:14,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48884.61 MB 2025-02-15 04:18:14,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10427.04 MB 2025-02-15 04:18:14,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44235.26 MB 2025-02-15 04:18:16,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:18:16,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:18:16,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:18:16,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23492.70 MB 2025-02-15 04:18:16,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24023.55 MB 2025-02-15 04:18:16,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:18:16,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48884.61 MB 2025-02-15 04:18:16,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 04:18:16,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19857.93 MB 2025-02-15 04:18:16,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28002.09 MB 2025-02-15 04:18:16,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:18:16,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:18:16,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:18:16,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-15 04:18:16,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25913.08 MB 2025-02-15 04:18:16,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:18:16,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 04:18:16,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 04:18:16,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:18:16,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27330.51 MB 2025-02-15 04:18:16,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:18:16,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:18:16,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:18:16,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25913.08 MB 2025-02-15 04:18:16,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-15 04:18:16,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:18:16,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 04:18:16,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36341.55 MB 2025-02-15 04:18:16,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 04:18:16,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-15 04:18:16,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:18:16,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:18:16,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:18:16,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-15 04:18:16,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-15 04:18:16,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:18:16,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 04:18:16,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36341.55 MB 2025-02-15 04:18:16,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 04:18:16,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-15 04:18:16,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:18:16,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:18:16,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:18:16,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.48 MB 2025-02-15 04:18:16,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30455.48 MB 2025-02-15 04:18:16,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:18:16,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36341.55 MB 2025-02-15 04:18:16,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36758.88 MB 2025-02-15 04:18:16,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:18:16,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31163.27 MB 2025-02-15 04:18:16,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:18:16,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:18:16,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:18:16,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30868.37 MB 2025-02-15 04:18:16,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31096.64 MB 2025-02-15 04:18:16,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-15 04:18:16,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36758.88 MB 2025-02-15 04:18:16,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36758.88 MB 2025-02-15 04:18:16,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:18:16,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31334.57 MB 2025-02-15 04:18:16,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:18:16,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:18:16,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.37 seconds 2025-02-15 04:18:16,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18139.08 MB 2025-02-15 04:18:16,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31297.49 MB 2025-02-15 04:18:16,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13158.41 MB 2025-02-15 04:18:16,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50453.28 MB 2025-02-15 04:18:16,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36758.88 MB 2025-02-15 04:18:16,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13694.40 MB 2025-02-15 04:18:16,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31334.57 MB 2025-02-15 04:18:16,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:18:16,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:18:16,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:18:16,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31297.49 MB 2025-02-15 04:18:16,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23130.42 MB 2025-02-15 04:18:16,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8167.07 MB 2025-02-15 04:18:16,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36758.88 MB 2025-02-15 04:18:16,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36758.88 MB 2025-02-15 04:18:16,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:18:16,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33798.10 MB 2025-02-15 04:18:16,758 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-15 04:18:16,758 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:18:16,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:18:16,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:18:16,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:18:16,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:18:16,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23130.42 MB 2025-02-15 04:18:16,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31531.95 MB 2025-02-15 04:18:16,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-15 04:18:16,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36758.88 MB 2025-02-15 04:18:16,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45113.93 MB 2025-02-15 04:18:16,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 04:18:16,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31531.95 MB 2025-02-15 04:18:16,923 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-15 04:18:16,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:18:16,924 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:18:16,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:18:16,925 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:18:16,930 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:18:16,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:18:16,931 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:18:16,931 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:19:39,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:39,636 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:19:39,642 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:19:39,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:39,646 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1071, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:19:39,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:39,647 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1071, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:19:56,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:19:56,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:19:56,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.44 seconds 2025-02-15 04:19:56,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:56,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20431.61 MB 2025-02-15 04:19:56,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24221.81 MB 2025-02-15 04:19:56,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3790.21 MB 2025-02-15 04:19:56,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53468.99 MB 2025-02-15 04:19:56,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28626.12 MB 2025-02-15 04:19:56,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24842.86 MB 2025-02-15 04:19:56,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33073.87 MB 2025-02-15 04:19:56,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:19:56,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:19:56,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:19:56,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:56,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24221.81 MB 2025-02-15 04:19:56,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21346.69 MB 2025-02-15 04:19:56,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2875.12 MB 2025-02-15 04:19:56,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28626.12 MB 2025-02-15 04:19:56,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37039.90 MB 2025-02-15 04:19:56,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8413.77 MB 2025-02-15 04:19:56,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34733.80 MB 2025-02-15 04:19:58,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:19:58,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:19:58,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 04:19:58,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21346.69 MB 2025-02-15 04:19:58,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21877.54 MB 2025-02-15 04:19:58,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:19:58,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37039.90 MB 2025-02-15 04:19:58,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26958.89 MB 2025-02-15 04:19:58,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10081.01 MB 2025-02-15 04:19:58,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25856.18 MB 2025-02-15 04:19:58,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:19:58,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:19:58,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:19:58,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21877.54 MB 2025-02-15 04:19:58,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23767.07 MB 2025-02-15 04:19:58,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:19:58,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26958.89 MB 2025-02-15 04:19:58,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27902.61 MB 2025-02-15 04:19:58,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:19:58,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25184.50 MB 2025-02-15 04:19:58,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:19:58,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:19:58,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:19:58,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23767.07 MB 2025-02-15 04:19:58,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.93 MB 2025-02-15 04:19:58,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:19:58,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27902.61 MB 2025-02-15 04:19:58,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:19:58,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:19:58,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31553.21 MB 2025-02-15 04:19:58,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:19:58,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:19:58,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:19:58,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21877.54 MB 2025-02-15 04:19:58,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.93 MB 2025-02-15 04:19:58,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:19:58,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26958.89 MB 2025-02-15 04:19:58,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:19:58,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:19:58,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31553.21 MB 2025-02-15 04:19:58,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:19:58,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:19:58,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:19:58,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27542.47 MB 2025-02-15 04:19:58,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28309.47 MB 2025-02-15 04:19:58,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:19:58,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:19:58,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 04:19:58,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:19:58,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29017.26 MB 2025-02-15 04:19:58,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:19:58,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:19:58,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:19:58,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28722.36 MB 2025-02-15 04:19:58,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28950.82 MB 2025-02-15 04:19:58,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-15 04:19:58,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 04:19:58,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 04:19:58,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:19:58,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29152.99 MB 2025-02-15 04:19:58,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:19:58,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:19:58,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.84 seconds 2025-02-15 04:19:58,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16700.16 MB 2025-02-15 04:19:58,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29151.89 MB 2025-02-15 04:19:58,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12451.73 MB 2025-02-15 04:19:58,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53468.99 MB 2025-02-15 04:19:58,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 04:19:58,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19488.83 MB 2025-02-15 04:19:58,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29152.99 MB 2025-02-15 04:19:58,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:19:58,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:19:58,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:19:58,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29151.89 MB 2025-02-15 04:19:58,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21704.55 MB 2025-02-15 04:19:58,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7447.34 MB 2025-02-15 04:19:58,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 04:19:58,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-15 04:19:58,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:19:58,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31663.56 MB 2025-02-15 04:19:58,776 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:19:58,776 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:19:58,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:19:58,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:19:58,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:19:58,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:19:58,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21704.55 MB 2025-02-15 04:19:58,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30143.57 MB 2025-02-15 04:19:58,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:19:58,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-15 04:19:58,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42370.86 MB 2025-02-15 04:19:58,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:19:58,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30143.57 MB 2025-02-15 04:19:58,950 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:19:58,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:58,951 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:19:58,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:58,952 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:19:58,957 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:19:58,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:19:58,958 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:19:58,958 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:20:09,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:09,259 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:20:09,264 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:20:09,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:09,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1901, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:20:09,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:09,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1901, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:20:38,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:20:38,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:20:38,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.59 seconds 2025-02-15 04:20:38,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:38,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26215.18 MB 2025-02-15 04:20:38,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32942.84 MB 2025-02-15 04:20:38,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6727.66 MB 2025-02-15 04:20:38,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54955.87 MB 2025-02-15 04:20:38,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39952.84 MB 2025-02-15 04:20:38,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15003.03 MB 2025-02-15 04:20:38,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41801.84 MB 2025-02-15 04:20:39,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:20:39,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:20:39,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:20:39,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:39,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32942.84 MB 2025-02-15 04:20:39,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25660.56 MB 2025-02-15 04:20:39,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7282.29 MB 2025-02-15 04:20:39,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39952.84 MB 2025-02-15 04:20:39,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53838.09 MB 2025-02-15 04:20:39,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13885.24 MB 2025-02-15 04:20:39,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51692.65 MB 2025-02-15 04:20:40,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:20:40,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:20:40,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:20:40,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:40,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25660.56 MB 2025-02-15 04:20:40,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26191.40 MB 2025-02-15 04:20:40,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:20:40,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53838.09 MB 2025-02-15 04:20:40,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-15 04:20:40,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23374.86 MB 2025-02-15 04:20:40,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30170.98 MB 2025-02-15 04:20:40,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:20:40,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:20:40,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:20:40,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:40,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-15 04:20:40,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28080.93 MB 2025-02-15 04:20:40,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:20:40,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 04:20:40,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32350.67 MB 2025-02-15 04:20:40,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 04:20:40,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29498.36 MB 2025-02-15 04:20:41,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:20:41,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:20:41,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:20:41,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28080.93 MB 2025-02-15 04:20:41,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-15 04:20:41,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:20:41,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32350.67 MB 2025-02-15 04:20:41,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38012.98 MB 2025-02-15 04:20:41,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:20:41,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-15 04:20:41,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:20:41,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:20:41,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:20:41,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-15 04:20:41,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-15 04:20:41,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:20:41,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 04:20:41,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38012.98 MB 2025-02-15 04:20:41,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 04:20:41,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-15 04:20:41,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:20:41,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:20:41,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:20:41,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31856.33 MB 2025-02-15 04:20:41,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32623.33 MB 2025-02-15 04:20:41,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:20:41,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38012.98 MB 2025-02-15 04:20:41,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 04:20:41,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:20:41,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33331.12 MB 2025-02-15 04:20:41,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:20:41,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:20:41,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:20:41,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33036.22 MB 2025-02-15 04:20:41,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33264.49 MB 2025-02-15 04:20:41,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-15 04:20:41,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 04:20:41,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 04:20:41,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:20:41,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33493.87 MB 2025-02-15 04:20:41,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:20:41,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:20:41,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.08 seconds 2025-02-15 04:20:41,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19591.94 MB 2025-02-15 04:20:41,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33465.35 MB 2025-02-15 04:20:41,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13873.40 MB 2025-02-15 04:20:41,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54955.87 MB 2025-02-15 04:20:41,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 04:20:41,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16529.75 MB 2025-02-15 04:20:41,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33493.87 MB 2025-02-15 04:20:41,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:20:41,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:20:41,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:20:41,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33465.35 MB 2025-02-15 04:20:41,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24583.28 MB 2025-02-15 04:20:41,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8882.06 MB 2025-02-15 04:20:41,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 04:20:41,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 04:20:41,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:20:41,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35965.95 MB 2025-02-15 04:20:41,639 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-15 04:20:41,639 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:20:41,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:20:41,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:20:41,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:20:41,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:20:41,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24583.28 MB 2025-02-15 04:20:41,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32984.81 MB 2025-02-15 04:20:41,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-15 04:20:41,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 04:20:41,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46781.17 MB 2025-02-15 04:20:41,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 04:20:41,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32984.81 MB 2025-02-15 04:20:41,803 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-15 04:20:41,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:41,804 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:20:41,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:41,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:20:41,810 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:20:41,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:20:41,811 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:20:41,811 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:21:34,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:34,292 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:21:34,297 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:21:34,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:34,301 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 212, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:21:34,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:34,302 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 212, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:21:37,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:21:37,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:21:37,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-15 04:21:37,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:37,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.96 MB 2025-02-15 04:21:37,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.21 MB 2025-02-15 04:21:37,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.26 MB 2025-02-15 04:21:37,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55136.22 MB 2025-02-15 04:21:37,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23253.22 MB 2025-02-15 04:21:37,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31883.00 MB 2025-02-15 04:21:37,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24143.82 MB 2025-02-15 04:21:37,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:21:37,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:21:37,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:21:37,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:37,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.21 MB 2025-02-15 04:21:37,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15440.32 MB 2025-02-15 04:21:37,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 244.11 MB 2025-02-15 04:21:37,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23253.22 MB 2025-02-15 04:21:37,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23253.22 MB 2025-02-15 04:21:37,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:37,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17935.26 MB 2025-02-15 04:21:38,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:21:38,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:21:38,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 04:21:38,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15440.32 MB 2025-02-15 04:21:38,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15699.10 MB 2025-02-15 04:21:38,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.79 MB 2025-02-15 04:21:38,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23253.22 MB 2025-02-15 04:21:38,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21615.35 MB 2025-02-15 04:21:38,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 04:21:38,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19694.90 MB 2025-02-15 04:21:38,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:21:38,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:21:38,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:21:38,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-15 04:21:38,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16619.96 MB 2025-02-15 04:21:38,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.92 MB 2025-02-15 04:21:38,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21615.35 MB 2025-02-15 04:21:38,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21615.35 MB 2025-02-15 04:21:38,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:38,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17310.96 MB 2025-02-15 04:21:38,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:21:38,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:21:38,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:21:38,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16619.96 MB 2025-02-15 04:21:38,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-15 04:21:38,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1092.94 MB 2025-02-15 04:21:38,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21615.35 MB 2025-02-15 04:21:38,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21615.35 MB 2025-02-15 04:21:38,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:38,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-15 04:21:38,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:21:38,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:21:38,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:21:38,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-15 04:21:38,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-15 04:21:38,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2013.86 MB 2025-02-15 04:21:38,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21615.35 MB 2025-02-15 04:21:38,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21615.35 MB 2025-02-15 04:21:38,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:38,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-15 04:21:38,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:21:38,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:21:38,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:21:38,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18460.50 MB 2025-02-15 04:21:38,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.42 MB 2025-02-15 04:21:38,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 373.91 MB 2025-02-15 04:21:38,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21615.35 MB 2025-02-15 04:21:38,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-15 04:21:38,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 203.42 MB 2025-02-15 04:21:38,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19182.02 MB 2025-02-15 04:21:38,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:21:38,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:21:38,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:21:38,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19035.71 MB 2025-02-15 04:21:38,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19244.89 MB 2025-02-15 04:21:38,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.19 MB 2025-02-15 04:21:38,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21818.77 MB 2025-02-15 04:21:38,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-15 04:21:38,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:38,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19289.60 MB 2025-02-15 04:21:38,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:21:38,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:21:38,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.45 seconds 2025-02-15 04:21:38,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:38,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-15 04:21:38,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19445.55 MB 2025-02-15 04:21:38,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5738.22 MB 2025-02-15 04:21:38,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55136.22 MB 2025-02-15 04:21:38,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-15 04:21:38,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33317.45 MB 2025-02-15 04:21:38,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19445.55 MB 2025-02-15 04:21:39,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:21:39,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:21:39,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:21:39,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:39,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14730.02 MB 2025-02-15 04:21:39,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.28 MB 2025-02-15 04:21:39,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.26 MB 2025-02-15 04:21:39,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21818.77 MB 2025-02-15 04:21:39,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-15 04:21:39,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:21:39,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18039.02 MB 2025-02-15 04:21:39,041 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 04:21:39,041 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:21:39,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:21:39,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:21:39,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:21:39,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:21:39,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.28 MB 2025-02-15 04:21:39,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26160.24 MB 2025-02-15 04:21:39,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-15 04:21:39,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21818.77 MB 2025-02-15 04:21:39,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32283.56 MB 2025-02-15 04:21:39,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 04:21:39,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26160.24 MB 2025-02-15 04:21:39,222 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 04:21:39,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:39,223 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:21:39,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:39,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:21:39,229 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:21:39,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:21:39,230 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:21:39,230 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:22:29,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:29,077 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:22:29,082 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:22:29,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:29,086 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:22:29,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:29,087 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:22:47,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:22:47,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:22:47,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.58 seconds 2025-02-15 04:22:47,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:47,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21407.15 MB 2025-02-15 04:22:47,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25693.73 MB 2025-02-15 04:22:47,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4286.58 MB 2025-02-15 04:22:47,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40655.39 MB 2025-02-15 04:22:47,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31237.08 MB 2025-02-15 04:22:47,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9418.31 MB 2025-02-15 04:22:47,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34502.40 MB 2025-02-15 04:22:47,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:22:47,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:22:47,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:22:47,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:47,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25693.73 MB 2025-02-15 04:22:47,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22073.46 MB 2025-02-15 04:22:47,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3620.26 MB 2025-02-15 04:22:47,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31237.08 MB 2025-02-15 04:22:47,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40965.77 MB 2025-02-15 04:22:47,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9728.69 MB 2025-02-15 04:22:47,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38226.75 MB 2025-02-15 04:22:49,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:22:49,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:22:49,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:22:49,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:49,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22073.46 MB 2025-02-15 04:22:49,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22604.30 MB 2025-02-15 04:22:49,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:22:49,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40965.77 MB 2025-02-15 04:22:49,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28366.08 MB 2025-02-15 04:22:49,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12599.69 MB 2025-02-15 04:22:49,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26582.85 MB 2025-02-15 04:22:49,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:22:49,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:22:49,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:22:49,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:49,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22604.30 MB 2025-02-15 04:22:49,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24493.84 MB 2025-02-15 04:22:49,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:22:49,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28366.08 MB 2025-02-15 04:22:49,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28366.08 MB 2025-02-15 04:22:49,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:22:49,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25911.27 MB 2025-02-15 04:22:49,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:22:49,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:22:49,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:22:49,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:49,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24493.84 MB 2025-02-15 04:22:49,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.34 MB 2025-02-15 04:22:49,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.50 MB 2025-02-15 04:22:49,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28366.08 MB 2025-02-15 04:22:49,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34500.25 MB 2025-02-15 04:22:49,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 04:22:49,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32280.62 MB 2025-02-15 04:22:49,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:22:49,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:22:49,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:22:49,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:49,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22604.30 MB 2025-02-15 04:22:49,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.34 MB 2025-02-15 04:22:49,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.04 MB 2025-02-15 04:22:49,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28366.08 MB 2025-02-15 04:22:49,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34500.25 MB 2025-02-15 04:22:49,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 04:22:49,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32280.62 MB 2025-02-15 04:22:50,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:22:50,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:22:50,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:22:50,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:50,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28269.88 MB 2025-02-15 04:22:50,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29036.88 MB 2025-02-15 04:22:50,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:22:50,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34500.25 MB 2025-02-15 04:22:50,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34917.58 MB 2025-02-15 04:22:50,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:22:50,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29744.67 MB 2025-02-15 04:22:50,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:22:50,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:22:50,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:22:50,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:50,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29449.77 MB 2025-02-15 04:22:50,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29677.43 MB 2025-02-15 04:22:50,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.66 MB 2025-02-15 04:22:50,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34917.58 MB 2025-02-15 04:22:50,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34917.58 MB 2025-02-15 04:22:50,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:22:50,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29906.37 MB 2025-02-15 04:22:50,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:22:50,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:22:50,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.02 seconds 2025-02-15 04:22:50,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:50,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17187.93 MB 2025-02-15 04:22:50,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29878.04 MB 2025-02-15 04:22:50,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12690.11 MB 2025-02-15 04:22:50,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40655.39 MB 2025-02-15 04:22:50,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34917.58 MB 2025-02-15 04:22:50,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5737.81 MB 2025-02-15 04:22:50,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29906.37 MB 2025-02-15 04:22:50,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:22:50,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:22:50,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:22:50,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:50,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29878.04 MB 2025-02-15 04:22:50,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22185.32 MB 2025-02-15 04:22:50,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7692.71 MB 2025-02-15 04:22:50,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34917.58 MB 2025-02-15 04:22:50,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34917.58 MB 2025-02-15 04:22:50,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:22:50,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.87 MB 2025-02-15 04:22:50,393 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 04:22:50,393 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:22:50,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:22:50,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:22:50,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:22:50,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:22:50,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22185.32 MB 2025-02-15 04:22:50,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30604.40 MB 2025-02-15 04:22:50,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 04:22:50,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34917.58 MB 2025-02-15 04:22:50,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43289.41 MB 2025-02-15 04:22:50,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 04:22:50,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30604.40 MB 2025-02-15 04:22:50,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 04:22:50,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:50,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:22:50,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:50,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:22:50,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:22:50,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:22:50,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:22:50,569 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:23:31,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:31,857 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:23:31,862 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:23:31,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:31,866 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:23:31,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:31,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:23:50,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:23:50,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:23:50,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.86 seconds 2025-02-15 04:23:50,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:50,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-15 04:23:50,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-15 04:23:50,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 04:23:50,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51661.24 MB 2025-02-15 04:23:50,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33342.62 MB 2025-02-15 04:23:50,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18318.62 MB 2025-02-15 04:23:50,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-15 04:23:50,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:23:50,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:23:50,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:23:50,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:50,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-15 04:23:50,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-15 04:23:50,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-15 04:23:50,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33342.62 MB 2025-02-15 04:23:50,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41892.71 MB 2025-02-15 04:23:50,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8550.09 MB 2025-02-15 04:23:50,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38612.19 MB 2025-02-15 04:23:52,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:23:52,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:23:52,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:23:52,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:52,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-15 04:23:52,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-15 04:23:52,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:23:52,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41892.71 MB 2025-02-15 04:23:52,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24849.15 MB 2025-02-15 04:23:52,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17043.55 MB 2025-02-15 04:23:52,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26615.08 MB 2025-02-15 04:23:52,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:23:52,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:23:52,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:23:52,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:52,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 04:23:52,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-15 04:23:52,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:23:52,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24849.15 MB 2025-02-15 04:23:52,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27680.31 MB 2025-02-15 04:23:52,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:23:52,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-15 04:23:52,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:23:52,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:23:52,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:23:52,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:52,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-15 04:23:52,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 04:23:52,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:23:52,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27680.31 MB 2025-02-15 04:23:52,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-15 04:23:52,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:23:52,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-15 04:23:52,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:23:52,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:23:52,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:23:52,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:52,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 04:23:52,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 04:23:52,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:23:52,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24849.15 MB 2025-02-15 04:23:52,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-15 04:23:52,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-15 04:23:52,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-15 04:23:53,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:23:53,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:23:53,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:23:53,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:53,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.43 MB 2025-02-15 04:23:53,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.43 MB 2025-02-15 04:23:53,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:23:53,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-15 04:23:53,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 04:23:53,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:23:53,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.22 MB 2025-02-15 04:23:53,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:23:53,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:23:53,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:23:53,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:53,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.32 MB 2025-02-15 04:23:53,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29709.07 MB 2025-02-15 04:23:53,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.75 MB 2025-02-15 04:23:53,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34703.67 MB 2025-02-15 04:23:53,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 04:23:53,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:23:53,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29945.33 MB 2025-02-15 04:23:53,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:23:53,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:23:53,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.29 seconds 2025-02-15 04:23:53,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:53,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-15 04:23:53,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29909.38 MB 2025-02-15 04:23:53,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.55 MB 2025-02-15 04:23:53,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51661.24 MB 2025-02-15 04:23:53,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 04:23:53,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16957.57 MB 2025-02-15 04:23:53,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29945.33 MB 2025-02-15 04:23:53,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:23:53,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:23:53,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:23:53,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:53,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29909.38 MB 2025-02-15 04:23:53,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22201.95 MB 2025-02-15 04:23:53,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7707.43 MB 2025-02-15 04:23:53,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34703.67 MB 2025-02-15 04:23:53,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 04:23:53,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:23:53,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32411.53 MB 2025-02-15 04:23:53,450 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 04:23:53,450 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:23:53,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:23:53,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:23:53,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:23:53,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:23:53,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22201.95 MB 2025-02-15 04:23:53,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30609.69 MB 2025-02-15 04:23:53,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 04:23:53,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34703.67 MB 2025-02-15 04:23:53,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45153.78 MB 2025-02-15 04:23:53,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10450.11 MB 2025-02-15 04:23:53,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30609.69 MB 2025-02-15 04:23:53,616 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 04:23:53,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:53,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:23:53,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:53,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:23:53,623 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:23:53,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:23:53,624 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:23:53,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:24:27,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:27,189 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:24:27,194 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:24:27,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:27,198 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1017, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:24:27,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:27,199 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1017, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:24:42,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:24:42,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:24:42,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.74 seconds 2025-02-15 04:24:42,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:42,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20055.33 MB 2025-02-15 04:24:42,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.43 MB 2025-02-15 04:24:42,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3599.11 MB 2025-02-15 04:24:42,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53513.03 MB 2025-02-15 04:24:42,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30545.02 MB 2025-02-15 04:24:42,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22968.01 MB 2025-02-15 04:24:42,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32471.10 MB 2025-02-15 04:24:43,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:24:43,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:24:43,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:24:43,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:43,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.43 MB 2025-02-15 04:24:43,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21064.92 MB 2025-02-15 04:24:43,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2589.51 MB 2025-02-15 04:24:43,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30545.02 MB 2025-02-15 04:24:43,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36805.02 MB 2025-02-15 04:24:43,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6260.00 MB 2025-02-15 04:24:43,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34138.11 MB 2025-02-15 04:24:44,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:24:44,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:24:44,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:24:44,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:44,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21064.92 MB 2025-02-15 04:24:44,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21595.76 MB 2025-02-15 04:24:44,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:24:44,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36805.02 MB 2025-02-15 04:24:44,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24853.35 MB 2025-02-15 04:24:44,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11951.67 MB 2025-02-15 04:24:44,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25575.34 MB 2025-02-15 04:24:44,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:24:44,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:24:44,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:24:44,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:44,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21595.76 MB 2025-02-15 04:24:44,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23485.29 MB 2025-02-15 04:24:44,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:24:44,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24853.35 MB 2025-02-15 04:24:44,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27684.50 MB 2025-02-15 04:24:44,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:24:44,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24902.72 MB 2025-02-15 04:24:45,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:24:45,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:24:45,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:24:45,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23485.29 MB 2025-02-15 04:24:45,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.15 MB 2025-02-15 04:24:45,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:24:45,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27684.50 MB 2025-02-15 04:24:45,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33346.81 MB 2025-02-15 04:24:45,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:24:45,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31271.43 MB 2025-02-15 04:24:45,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:24:45,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:24:45,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:24:45,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21595.76 MB 2025-02-15 04:24:45,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.15 MB 2025-02-15 04:24:45,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:24:45,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24853.35 MB 2025-02-15 04:24:45,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33346.81 MB 2025-02-15 04:24:45,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 04:24:45,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31271.43 MB 2025-02-15 04:24:45,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:24:45,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:24:45,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:24:45,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27260.69 MB 2025-02-15 04:24:45,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28027.69 MB 2025-02-15 04:24:45,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:24:45,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33346.81 MB 2025-02-15 04:24:45,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-15 04:24:45,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:24:45,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28735.48 MB 2025-02-15 04:24:45,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:24:45,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:24:45,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:24:45,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28440.58 MB 2025-02-15 04:24:45,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28667.96 MB 2025-02-15 04:24:45,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.38 MB 2025-02-15 04:24:45,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33759.95 MB 2025-02-15 04:24:45,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-15 04:24:45,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:24:45,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28846.42 MB 2025-02-15 04:24:45,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:24:45,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:24:45,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.16 seconds 2025-02-15 04:24:45,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16512.02 MB 2025-02-15 04:24:45,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28869.03 MB 2025-02-15 04:24:45,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12357.01 MB 2025-02-15 04:24:45,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53513.03 MB 2025-02-15 04:24:45,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-15 04:24:45,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19753.07 MB 2025-02-15 04:24:45,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.03 MB 2025-02-15 04:24:45,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:24:45,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:24:45,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:24:45,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28869.03 MB 2025-02-15 04:24:45,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21516.41 MB 2025-02-15 04:24:45,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7352.63 MB 2025-02-15 04:24:45,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33759.95 MB 2025-02-15 04:24:45,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-15 04:24:45,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:24:45,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31380.70 MB 2025-02-15 04:24:45,645 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:24:45,646 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:24:45,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:24:45,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:24:45,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:24:45,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:24:45,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21516.41 MB 2025-02-15 04:24:45,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29955.43 MB 2025-02-15 04:24:45,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:24:45,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33759.95 MB 2025-02-15 04:24:45,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42150.66 MB 2025-02-15 04:24:45,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:24:45,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29955.43 MB 2025-02-15 04:24:45,811 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:24:45,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:45,812 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:24:45,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:45,813 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:24:45,818 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:24:45,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:24:45,819 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:24:45,819 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:26:38,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:38,921 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:26:38,926 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:26:38,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:38,930 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 615, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:26:38,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:38,931 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 615, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:26:48,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:26:48,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:26:48,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.41 seconds 2025-02-15 04:26:48,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:48,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17254.12 MB 2025-02-15 04:26:48,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19430.97 MB 2025-02-15 04:26:48,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2176.84 MB 2025-02-15 04:26:48,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54735.67 MB 2025-02-15 04:26:48,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22842.18 MB 2025-02-15 04:26:48,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31893.49 MB 2025-02-15 04:26:48,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28310.94 MB 2025-02-15 04:26:48,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:26:48,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:26:48,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:26:48,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:48,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19430.97 MB 2025-02-15 04:26:48,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18975.04 MB 2025-02-15 04:26:48,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -455.92 MB 2025-02-15 04:26:48,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22842.18 MB 2025-02-15 04:26:48,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28577.89 MB 2025-02-15 04:26:48,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5735.71 MB 2025-02-15 04:26:48,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27975.89 MB 2025-02-15 04:26:50,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:26:50,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:26:50,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 04:26:50,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18975.04 MB 2025-02-15 04:26:50,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19505.89 MB 2025-02-15 04:26:50,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:26:50,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28577.89 MB 2025-02-15 04:26:50,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22080.91 MB 2025-02-15 04:26:50,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6496.98 MB 2025-02-15 04:26:50,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23485.47 MB 2025-02-15 04:26:50,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:26:50,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:26:50,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:26:50,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19505.89 MB 2025-02-15 04:26:50,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21395.42 MB 2025-02-15 04:26:50,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:26:50,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22080.91 MB 2025-02-15 04:26:50,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23968.35 MB 2025-02-15 04:26:50,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 04:26:50,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-15 04:26:50,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:26:50,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:26:50,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:26:50,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21395.42 MB 2025-02-15 04:26:50,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23637.28 MB 2025-02-15 04:26:50,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:26:50,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23968.35 MB 2025-02-15 04:26:50,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31046.24 MB 2025-02-15 04:26:50,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 04:26:50,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29181.56 MB 2025-02-15 04:26:50,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:26:50,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:26:50,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:26:50,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19505.89 MB 2025-02-15 04:26:50,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23637.28 MB 2025-02-15 04:26:50,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:26:50,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22080.91 MB 2025-02-15 04:26:50,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31046.24 MB 2025-02-15 04:26:50,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 04:26:50,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29181.56 MB 2025-02-15 04:26:50,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:26:50,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:26:50,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:26:50,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25170.82 MB 2025-02-15 04:26:50,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25937.82 MB 2025-02-15 04:26:50,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:26:50,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31046.24 MB 2025-02-15 04:26:50,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31461.47 MB 2025-02-15 04:26:50,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:26:50,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26645.61 MB 2025-02-15 04:26:50,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:26:50,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:26:50,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:26:50,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26350.71 MB 2025-02-15 04:26:50,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26577.51 MB 2025-02-15 04:26:50,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.80 MB 2025-02-15 04:26:50,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31461.47 MB 2025-02-15 04:26:50,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31461.47 MB 2025-02-15 04:26:50,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:26:50,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26797.98 MB 2025-02-15 04:26:50,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:26:50,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:26:50,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.78 seconds 2025-02-15 04:26:50,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15111.42 MB 2025-02-15 04:26:50,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26778.36 MB 2025-02-15 04:26:50,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11666.94 MB 2025-02-15 04:26:50,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54735.67 MB 2025-02-15 04:26:50,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31461.47 MB 2025-02-15 04:26:50,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23274.19 MB 2025-02-15 04:26:50,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26797.98 MB 2025-02-15 04:26:50,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:26:50,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:26:50,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:26:50,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:50,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26778.36 MB 2025-02-15 04:26:50,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20104.89 MB 2025-02-15 04:26:50,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6673.46 MB 2025-02-15 04:26:50,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31461.47 MB 2025-02-15 04:26:50,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31461.47 MB 2025-02-15 04:26:50,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:26:50,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29280.81 MB 2025-02-15 04:26:51,000 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 04:26:51,000 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:26:51,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:26:51,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:26:51,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:26:51,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:26:51,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20104.89 MB 2025-02-15 04:26:51,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28514.19 MB 2025-02-15 04:26:51,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 04:26:51,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31461.47 MB 2025-02-15 04:26:51,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41911.58 MB 2025-02-15 04:26:51,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10450.11 MB 2025-02-15 04:26:51,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28514.19 MB 2025-02-15 04:26:51,164 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 04:26:51,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:51,166 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:26:51,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:51,167 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:26:51,171 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:26:51,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:26:51,172 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:26:51,172 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:27:02,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:02,126 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:27:02,131 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:27:02,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:02,134 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2418, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:27:02,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:02,135 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2418, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:27:39,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:27:39,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:27:39,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.45 seconds 2025-02-15 04:27:39,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:39,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29817.72 MB 2025-02-15 04:27:39,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38374.88 MB 2025-02-15 04:27:39,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8557.17 MB 2025-02-15 04:27:39,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58831.41 MB 2025-02-15 04:27:39,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41955.62 MB 2025-02-15 04:27:39,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16875.78 MB 2025-02-15 04:27:39,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47216.32 MB 2025-02-15 04:27:39,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:27:39,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:27:39,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:27:39,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:39,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38374.88 MB 2025-02-15 04:27:39,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28349.33 MB 2025-02-15 04:27:39,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10025.56 MB 2025-02-15 04:27:39,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41955.62 MB 2025-02-15 04:27:39,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60083.40 MB 2025-02-15 04:27:39,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18127.78 MB 2025-02-15 04:27:39,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62438.79 MB 2025-02-15 04:27:41,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:27:41,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:27:41,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:27:41,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:41,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28349.33 MB 2025-02-15 04:27:41,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28880.17 MB 2025-02-15 04:27:41,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:27:41,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60083.40 MB 2025-02-15 04:27:41,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31241.27 MB 2025-02-15 04:27:41,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28842.13 MB 2025-02-15 04:27:41,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32858.72 MB 2025-02-15 04:27:41,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:27:41,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:27:41,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:27:41,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:41,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28880.17 MB 2025-02-15 04:27:41,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30769.31 MB 2025-02-15 04:27:41,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.14 MB 2025-02-15 04:27:41,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31241.27 MB 2025-02-15 04:27:41,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34072.43 MB 2025-02-15 04:27:41,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 04:27:41,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32186.74 MB 2025-02-15 04:27:41,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:27:41,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:27:41,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:27:41,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:41,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30769.31 MB 2025-02-15 04:27:41,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33011.17 MB 2025-02-15 04:27:41,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:27:41,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34072.43 MB 2025-02-15 04:27:41,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40206.60 MB 2025-02-15 04:27:41,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 04:27:41,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38555.45 MB 2025-02-15 04:27:41,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:27:41,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:27:41,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:27:41,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:41,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28880.17 MB 2025-02-15 04:27:41,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33011.17 MB 2025-02-15 04:27:41,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.00 MB 2025-02-15 04:27:41,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31241.27 MB 2025-02-15 04:27:41,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40206.60 MB 2025-02-15 04:27:41,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 04:27:41,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38555.45 MB 2025-02-15 04:27:42,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:27:42,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:27:42,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:27:42,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:42,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34544.71 MB 2025-02-15 04:27:42,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35311.71 MB 2025-02-15 04:27:42,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:27:42,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40206.60 MB 2025-02-15 04:27:42,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40623.93 MB 2025-02-15 04:27:42,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:27:42,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36019.50 MB 2025-02-15 04:27:42,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:27:42,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:27:42,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:27:42,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:42,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35724.60 MB 2025-02-15 04:27:42,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35952.77 MB 2025-02-15 04:27:42,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 04:27:42,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40623.93 MB 2025-02-15 04:27:42,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40623.93 MB 2025-02-15 04:27:42,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:27:42,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36166.76 MB 2025-02-15 04:27:42,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:27:42,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:27:42,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.03 seconds 2025-02-15 04:27:42,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:42,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-15 04:27:42,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36152.86 MB 2025-02-15 04:27:42,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14759.65 MB 2025-02-15 04:27:42,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54551.12 MB 2025-02-15 04:27:42,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40623.93 MB 2025-02-15 04:27:42,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13927.19 MB 2025-02-15 04:27:42,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36166.76 MB 2025-02-15 04:27:42,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:27:42,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:27:42,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:27:42,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:42,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36152.86 MB 2025-02-15 04:27:42,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26383.13 MB 2025-02-15 04:27:42,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9769.74 MB 2025-02-15 04:27:42,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40623.93 MB 2025-02-15 04:27:42,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40623.93 MB 2025-02-15 04:27:42,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:27:42,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38652.24 MB 2025-02-15 04:27:42,458 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 04:27:42,458 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:27:42,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:27:42,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:27:42,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:27:42,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:27:42,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26383.13 MB 2025-02-15 04:27:42,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34780.53 MB 2025-02-15 04:27:42,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 04:27:42,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40623.93 MB 2025-02-15 04:27:42,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44799.36 MB 2025-02-15 04:27:42,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 04:27:42,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.53 MB 2025-02-15 04:27:42,621 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 04:27:42,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:42,623 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:27:42,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:42,624 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:27:42,628 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:27:42,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:27:42,629 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:27:42,629 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:28:02,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:02,051 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:28:02,056 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:28:02,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:02,059 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:28:02,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:02,060 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:28:05,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:28:05,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:28:05,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.59 seconds 2025-02-15 04:28:05,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-15 04:28:05,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15395.85 MB 2025-02-15 04:28:05,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.50 MB 2025-02-15 04:28:05,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53150.22 MB 2025-02-15 04:28:05,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20241.71 MB 2025-02-15 04:28:05,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32908.51 MB 2025-02-15 04:28:05,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24276.21 MB 2025-02-15 04:28:05,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:28:05,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:28:05,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:28:05,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15395.85 MB 2025-02-15 04:28:05,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14274.95 MB 2025-02-15 04:28:05,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1120.90 MB 2025-02-15 04:28:05,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20241.71 MB 2025-02-15 04:28:05,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20241.71 MB 2025-02-15 04:28:05,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15609.42 MB 2025-02-15 04:28:05,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:28:05,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:28:05,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:28:05,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14274.95 MB 2025-02-15 04:28:05,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14294.86 MB 2025-02-15 04:28:05,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.91 MB 2025-02-15 04:28:05,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20241.71 MB 2025-02-15 04:28:05,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 04:28:05,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 04:28:05,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15232.31 MB 2025-02-15 04:28:05,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:28:05,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:28:05,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:28:05,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14294.79 MB 2025-02-15 04:28:05,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14365.63 MB 2025-02-15 04:28:05,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 70.84 MB 2025-02-15 04:28:05,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 04:28:05,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 04:28:05,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14418.79 MB 2025-02-15 04:28:05,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:28:05,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:28:05,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:28:05,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14365.63 MB 2025-02-15 04:28:05,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14449.81 MB 2025-02-15 04:28:05,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 84.18 MB 2025-02-15 04:28:05,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 04:28:05,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 04:28:05,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14657.66 MB 2025-02-15 04:28:05,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:28:05,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:28:05,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:28:05,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14294.79 MB 2025-02-15 04:28:05,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14449.81 MB 2025-02-15 04:28:05,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 155.02 MB 2025-02-15 04:28:05,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 04:28:05,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 04:28:05,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14657.66 MB 2025-02-15 04:28:05,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:28:05,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:28:05,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:28:05,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14507.32 MB 2025-02-15 04:28:05,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14536.09 MB 2025-02-15 04:28:05,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 28.76 MB 2025-02-15 04:28:05,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 04:28:05,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18614.32 MB 2025-02-15 04:28:05,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10.49 MB 2025-02-15 04:28:05,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.95 MB 2025-02-15 04:28:05,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:28:05,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:28:05,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:28:05,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14551.58 MB 2025-02-15 04:28:05,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14571.80 MB 2025-02-15 04:28:05,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 20.22 MB 2025-02-15 04:28:05,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18614.32 MB 2025-02-15 04:28:05,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18614.32 MB 2025-02-15 04:28:05,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14571.80 MB 2025-02-15 04:28:05,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:28:05,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:28:05,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-15 04:28:05,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13773.53 MB 2025-02-15 04:28:05,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14608.61 MB 2025-02-15 04:28:05,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.09 MB 2025-02-15 04:28:05,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53150.22 MB 2025-02-15 04:28:05,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18614.32 MB 2025-02-15 04:28:05,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34535.90 MB 2025-02-15 04:28:05,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14608.61 MB 2025-02-15 04:28:05,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:28:05,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:28:05,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:28:05,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14608.61 MB 2025-02-15 04:28:05,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15160.52 MB 2025-02-15 04:28:05,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 551.91 MB 2025-02-15 04:28:05,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18614.32 MB 2025-02-15 04:28:05,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18618.52 MB 2025-02-15 04:28:05,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 04:28:05,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15215.70 MB 2025-02-15 04:28:05,851 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1483, cut from 1485 2025-02-15 04:28:05,852 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:28:05,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:28:05,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:28:05,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:28:05,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:28:05,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14416.44 MB 2025-02-15 04:28:05,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15961.95 MB 2025-02-15 04:28:05,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1545.51 MB 2025-02-15 04:28:05,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18618.52 MB 2025-02-15 04:28:05,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18618.52 MB 2025-02-15 04:28:05,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:28:05,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15961.95 MB 2025-02-15 04:28:05,882 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1275] 2025-02-15 04:28:05,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:05,884 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:28:05,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:05,885 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:28:05,889 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:28:05,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:28:05,890 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:28:05,891 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:29:03,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:03,930 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:29:03,935 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:29:03,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:03,938 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 355, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:29:03,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:03,939 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 355, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:29:09,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:29:09,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:29:09,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.43 seconds 2025-02-15 04:29:09,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:09,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15442.40 MB 2025-02-15 04:29:09,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16698.73 MB 2025-02-15 04:29:09,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.33 MB 2025-02-15 04:29:09,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20942.16 MB 2025-02-15 04:29:09,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18725.47 MB 2025-02-15 04:29:09,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2216.69 MB 2025-02-15 04:29:09,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25593.25 MB 2025-02-15 04:29:09,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:29:09,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:29:09,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:29:09,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:09,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16698.73 MB 2025-02-15 04:29:09,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17300.33 MB 2025-02-15 04:29:09,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.60 MB 2025-02-15 04:29:09,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18725.47 MB 2025-02-15 04:29:09,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23077.06 MB 2025-02-15 04:29:09,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4351.59 MB 2025-02-15 04:29:09,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21671.68 MB 2025-02-15 04:29:11,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:29:11,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:29:11,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-15 04:29:11,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17300.33 MB 2025-02-15 04:29:11,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17770.12 MB 2025-02-15 04:29:11,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 469.79 MB 2025-02-15 04:29:11,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23077.06 MB 2025-02-15 04:29:11,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19878.90 MB 2025-02-15 04:29:11,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3198.16 MB 2025-02-15 04:29:11,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21725.82 MB 2025-02-15 04:29:11,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:29:11,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:29:11,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:29:11,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17770.12 MB 2025-02-15 04:29:11,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19442.08 MB 2025-02-15 04:29:11,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1671.95 MB 2025-02-15 04:29:11,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19878.90 MB 2025-02-15 04:29:11,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22808.63 MB 2025-02-15 04:29:11,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2929.72 MB 2025-02-15 04:29:11,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20696.50 MB 2025-02-15 04:29:11,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:29:11,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:29:11,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:29:11,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19442.08 MB 2025-02-15 04:29:11,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21426.13 MB 2025-02-15 04:29:11,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1984.05 MB 2025-02-15 04:29:11,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22808.63 MB 2025-02-15 04:29:11,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28248.64 MB 2025-02-15 04:29:11,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5440.01 MB 2025-02-15 04:29:11,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26332.81 MB 2025-02-15 04:29:11,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:29:11,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:29:11,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:29:11,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17770.12 MB 2025-02-15 04:29:11,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21426.13 MB 2025-02-15 04:29:11,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3656.01 MB 2025-02-15 04:29:11,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19878.90 MB 2025-02-15 04:29:11,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28248.64 MB 2025-02-15 04:29:11,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-15 04:29:11,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26332.81 MB 2025-02-15 04:29:11,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:29:11,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:29:11,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:29:11,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22783.31 MB 2025-02-15 04:29:11,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23462.11 MB 2025-02-15 04:29:11,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 678.80 MB 2025-02-15 04:29:11,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28248.64 MB 2025-02-15 04:29:11,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28613.54 MB 2025-02-15 04:29:11,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 364.90 MB 2025-02-15 04:29:11,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.50 MB 2025-02-15 04:29:11,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:29:11,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:29:11,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:29:11,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23827.52 MB 2025-02-15 04:29:11,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24033.65 MB 2025-02-15 04:29:11,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.13 MB 2025-02-15 04:29:11,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28613.54 MB 2025-02-15 04:29:11,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 04:29:11,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 04:29:11,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24205.73 MB 2025-02-15 04:29:11,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:29:11,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:29:11,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.54 seconds 2025-02-15 04:29:11,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14205.55 MB 2025-02-15 04:29:11,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24234.72 MB 2025-02-15 04:29:11,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10029.17 MB 2025-02-15 04:29:11,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20942.16 MB 2025-02-15 04:29:11,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 04:29:11,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7675.58 MB 2025-02-15 04:29:11,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24234.72 MB 2025-02-15 04:29:11,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:29:11,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:29:11,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:29:11,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24234.72 MB 2025-02-15 04:29:11,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18992.33 MB 2025-02-15 04:29:11,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5242.39 MB 2025-02-15 04:29:11,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28617.74 MB 2025-02-15 04:29:11,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 04:29:11,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:29:11,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27449.65 MB 2025-02-15 04:29:11,764 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:29:11,764 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:29:11,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:29:11,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:29:11,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:29:11,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:11,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18992.33 MB 2025-02-15 04:29:11,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27431.36 MB 2025-02-15 04:29:11,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:29:11,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28617.74 MB 2025-02-15 04:29:11,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39107.69 MB 2025-02-15 04:29:11,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:29:11,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27431.36 MB 2025-02-15 04:29:11,928 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:29:11,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:11,930 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:29:11,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:11,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:29:11,936 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:29:11,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:11,937 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:29:11,937 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:29:24,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:24,777 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:29:24,782 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:29:24,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:24,785 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1077, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:29:24,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:24,786 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1077, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:29:41,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:29:41,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:29:41,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.64 seconds 2025-02-15 04:29:41,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:41,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-15 04:29:41,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24284.86 MB 2025-02-15 04:29:41,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.44 MB 2025-02-15 04:29:41,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51692.70 MB 2025-02-15 04:29:41,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26585.60 MB 2025-02-15 04:29:41,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25107.10 MB 2025-02-15 04:29:41,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33116.49 MB 2025-02-15 04:29:41,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:29:41,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:29:41,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:29:41,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:41,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24284.86 MB 2025-02-15 04:29:41,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21377.89 MB 2025-02-15 04:29:41,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2906.97 MB 2025-02-15 04:29:41,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26585.60 MB 2025-02-15 04:29:41,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35940.99 MB 2025-02-15 04:29:41,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9355.40 MB 2025-02-15 04:29:41,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35737.38 MB 2025-02-15 04:29:43,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:29:43,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:29:43,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:29:43,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21377.89 MB 2025-02-15 04:29:43,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21908.73 MB 2025-02-15 04:29:43,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:29:43,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35940.99 MB 2025-02-15 04:29:43,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 04:29:43,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11043.60 MB 2025-02-15 04:29:43,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25889.35 MB 2025-02-15 04:29:43,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:29:43,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:29:43,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:29:43,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.73 MB 2025-02-15 04:29:43,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23798.26 MB 2025-02-15 04:29:43,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:29:43,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 04:29:43,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26784.83 MB 2025-02-15 04:29:43,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 04:29:43,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25215.69 MB 2025-02-15 04:29:43,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:29:43,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:29:43,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:29:43,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23798.26 MB 2025-02-15 04:29:43,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26040.12 MB 2025-02-15 04:29:43,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:29:43,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26784.83 MB 2025-02-15 04:29:43,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-15 04:29:43,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:29:43,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31584.40 MB 2025-02-15 04:29:43,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:29:43,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:29:43,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:29:43,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.73 MB 2025-02-15 04:29:43,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26040.12 MB 2025-02-15 04:29:43,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:29:43,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 04:29:43,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-15 04:29:43,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 04:29:43,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31584.40 MB 2025-02-15 04:29:43,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:29:43,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:29:43,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:29:43,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27573.66 MB 2025-02-15 04:29:43,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28340.66 MB 2025-02-15 04:29:43,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:29:43,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33390.85 MB 2025-02-15 04:29:43,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 04:29:43,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 04:29:43,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29048.45 MB 2025-02-15 04:29:43,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:29:43,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:29:43,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:29:43,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28753.55 MB 2025-02-15 04:29:43,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28981.55 MB 2025-02-15 04:29:43,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-15 04:29:43,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 04:29:43,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 04:29:43,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:29:43,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29199.64 MB 2025-02-15 04:29:43,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:29:43,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:29:43,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.09 seconds 2025-02-15 04:29:43,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:43,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16721.06 MB 2025-02-15 04:29:43,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29182.06 MB 2025-02-15 04:29:43,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12461.00 MB 2025-02-15 04:29:43,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51692.70 MB 2025-02-15 04:29:43,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 04:29:43,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17886.61 MB 2025-02-15 04:29:43,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29199.64 MB 2025-02-15 04:29:44,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:29:44,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:29:44,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:29:44,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:44,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29182.06 MB 2025-02-15 04:29:44,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21717.03 MB 2025-02-15 04:29:44,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7465.03 MB 2025-02-15 04:29:44,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 04:29:44,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 04:29:44,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:29:44,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31687.01 MB 2025-02-15 04:29:44,162 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 04:29:44,162 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:29:44,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:29:44,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:29:44,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:29:44,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:29:44,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21717.03 MB 2025-02-15 04:29:44,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30131.98 MB 2025-02-15 04:29:44,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 04:29:44,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 04:29:44,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44266.68 MB 2025-02-15 04:29:44,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-15 04:29:44,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30131.98 MB 2025-02-15 04:29:44,325 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 04:29:44,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:44,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:29:44,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:44,327 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:29:44,332 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:29:44,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:29:44,333 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:29:44,333 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:30:37,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:37,198 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:30:37,205 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:30:37,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:37,211 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:30:37,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:37,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:30:41,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:30:41,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:30:41,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.50 seconds 2025-02-15 04:30:41,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:41,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14989.47 MB 2025-02-15 04:30:41,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16015.77 MB 2025-02-15 04:30:41,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-15 04:30:41,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52634.32 MB 2025-02-15 04:30:41,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20713.57 MB 2025-02-15 04:30:41,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31920.75 MB 2025-02-15 04:30:41,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24913.83 MB 2025-02-15 04:30:41,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:30:41,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:30:41,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:30:41,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:41,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16015.77 MB 2025-02-15 04:30:41,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16400.57 MB 2025-02-15 04:30:41,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 384.80 MB 2025-02-15 04:30:41,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20713.57 MB 2025-02-15 04:30:41,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22642.95 MB 2025-02-15 04:30:41,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1929.38 MB 2025-02-15 04:30:41,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19864.38 MB 2025-02-15 04:30:43,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:30:43,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:30:43,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.31 seconds 2025-02-15 04:30:43,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16400.57 MB 2025-02-15 04:30:43,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16764.20 MB 2025-02-15 04:30:43,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 363.63 MB 2025-02-15 04:30:43,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22642.95 MB 2025-02-15 04:30:43,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20698.89 MB 2025-02-15 04:30:43,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1944.06 MB 2025-02-15 04:30:43,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20741.13 MB 2025-02-15 04:30:43,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:30:43,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:30:43,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:30:43,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-15 04:30:43,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18058.23 MB 2025-02-15 04:30:43,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.03 MB 2025-02-15 04:30:43,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-15 04:30:43,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20698.89 MB 2025-02-15 04:30:43,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:30:43,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.17 MB 2025-02-15 04:30:43,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:30:43,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:30:43,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:30:43,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18058.23 MB 2025-02-15 04:30:43,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-15 04:30:43,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1535.69 MB 2025-02-15 04:30:43,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-15 04:30:43,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24912.07 MB 2025-02-15 04:30:43,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4213.18 MB 2025-02-15 04:30:43,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-15 04:30:43,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:30:43,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:30:43,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 04:30:43,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-15 04:30:43,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-15 04:30:43,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2829.73 MB 2025-02-15 04:30:43,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-15 04:30:43,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24912.07 MB 2025-02-15 04:30:43,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4213.18 MB 2025-02-15 04:30:43,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-15 04:30:43,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:30:43,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:30:43,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:30:43,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20644.40 MB 2025-02-15 04:30:43,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21169.79 MB 2025-02-15 04:30:43,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 525.40 MB 2025-02-15 04:30:43,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24912.07 MB 2025-02-15 04:30:43,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25195.18 MB 2025-02-15 04:30:43,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 283.12 MB 2025-02-15 04:30:43,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21654.63 MB 2025-02-15 04:30:43,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:30:43,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:30:43,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:30:43,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21452.63 MB 2025-02-15 04:30:43,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21658.58 MB 2025-02-15 04:30:43,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.96 MB 2025-02-15 04:30:43,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25195.18 MB 2025-02-15 04:30:43,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25199.38 MB 2025-02-15 04:30:43,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 04:30:43,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21753.51 MB 2025-02-15 04:30:43,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:30:43,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:30:43,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.12 seconds 2025-02-15 04:30:43,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-15 04:30:43,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21859.66 MB 2025-02-15 04:30:43,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7880.57 MB 2025-02-15 04:30:43,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52634.32 MB 2025-02-15 04:30:43,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25199.38 MB 2025-02-15 04:30:43,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27434.94 MB 2025-02-15 04:30:43,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21859.66 MB 2025-02-15 04:30:43,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:30:43,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:30:43,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:30:43,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21859.66 MB 2025-02-15 04:30:43,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24873.69 MB 2025-02-15 04:30:43,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:30:43,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25199.38 MB 2025-02-15 04:30:43,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26273.12 MB 2025-02-15 04:30:43,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-15 04:30:43,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25175.32 MB 2025-02-15 04:30:43,777 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:30:43,778 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:30:43,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:30:43,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:30:43,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:30:43,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:30:43,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18388.32 MB 2025-02-15 04:30:43,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26827.35 MB 2025-02-15 04:30:43,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:30:43,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26273.12 MB 2025-02-15 04:30:43,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-15 04:30:43,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:30:43,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26827.35 MB 2025-02-15 04:30:43,943 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:30:43,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:43,944 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:30:43,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:43,945 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:30:43,950 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:30:43,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:43,951 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:30:43,951 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:30:52,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:52,692 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:30:52,696 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:30:52,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:52,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1011, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:30:52,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:30:52,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1011, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:31:08,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:31:08,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:31:08,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.57 seconds 2025-02-15 04:31:08,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:08,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20013.52 MB 2025-02-15 04:31:08,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23591.39 MB 2025-02-15 04:31:08,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3577.87 MB 2025-02-15 04:31:08,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47248.83 MB 2025-02-15 04:31:08,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28449.96 MB 2025-02-15 04:31:08,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18798.87 MB 2025-02-15 04:31:08,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32429.29 MB 2025-02-15 04:31:08,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:31:08,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:31:08,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:31:08,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:08,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23591.39 MB 2025-02-15 04:31:08,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21034.77 MB 2025-02-15 04:31:08,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2556.62 MB 2025-02-15 04:31:08,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28449.96 MB 2025-02-15 04:31:08,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37459.33 MB 2025-02-15 04:31:08,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9009.36 MB 2025-02-15 04:31:08,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34700.57 MB 2025-02-15 04:31:10,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:31:10,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:31:10,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:31:10,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21034.77 MB 2025-02-15 04:31:10,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21565.62 MB 2025-02-15 04:31:10,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:31:10,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37459.33 MB 2025-02-15 04:31:10,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26994.54 MB 2025-02-15 04:31:10,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10464.79 MB 2025-02-15 04:31:10,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25544.16 MB 2025-02-15 04:31:10,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:31:10,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:31:10,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:31:10,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21565.62 MB 2025-02-15 04:31:10,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23455.15 MB 2025-02-15 04:31:10,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:31:10,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26994.54 MB 2025-02-15 04:31:10,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27938.26 MB 2025-02-15 04:31:10,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:31:10,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24872.58 MB 2025-02-15 04:31:10,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:31:10,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:31:10,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:31:10,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23455.15 MB 2025-02-15 04:31:10,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25697.01 MB 2025-02-15 04:31:10,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:31:10,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27938.26 MB 2025-02-15 04:31:10,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33128.71 MB 2025-02-15 04:31:10,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 04:31:10,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31241.29 MB 2025-02-15 04:31:10,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:31:10,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:31:10,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:31:10,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21565.62 MB 2025-02-15 04:31:10,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25697.01 MB 2025-02-15 04:31:10,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:31:10,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26994.54 MB 2025-02-15 04:31:10,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33128.71 MB 2025-02-15 04:31:10,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 04:31:10,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31241.29 MB 2025-02-15 04:31:10,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:31:10,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:31:10,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:31:10,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27230.55 MB 2025-02-15 04:31:10,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27997.55 MB 2025-02-15 04:31:10,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:31:10,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33128.71 MB 2025-02-15 04:31:10,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33541.85 MB 2025-02-15 04:31:10,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 04:31:10,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28705.34 MB 2025-02-15 04:31:10,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:31:10,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:31:10,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:31:10,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28410.44 MB 2025-02-15 04:31:10,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28638.61 MB 2025-02-15 04:31:10,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 04:31:10,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33541.85 MB 2025-02-15 04:31:10,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33541.85 MB 2025-02-15 04:31:10,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:10,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28858.04 MB 2025-02-15 04:31:10,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:31:10,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:31:10,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.98 seconds 2025-02-15 04:31:10,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16491.11 MB 2025-02-15 04:31:10,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28839.46 MB 2025-02-15 04:31:10,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12348.35 MB 2025-02-15 04:31:10,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47248.83 MB 2025-02-15 04:31:10,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33541.85 MB 2025-02-15 04:31:10,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13706.99 MB 2025-02-15 04:31:10,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28858.04 MB 2025-02-15 04:31:10,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:31:10,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:31:10,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:31:10,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18481.54 MB 2025-02-15 04:31:10,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21486.73 MB 2025-02-15 04:31:10,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.19 MB 2025-02-15 04:31:10,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33541.85 MB 2025-02-15 04:31:10,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33541.85 MB 2025-02-15 04:31:10,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:10,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21787.21 MB 2025-02-15 04:31:10,965 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 04:31:10,965 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:31:10,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:31:10,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:31:10,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:31:10,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:10,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21486.73 MB 2025-02-15 04:31:10,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29900.71 MB 2025-02-15 04:31:10,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 04:31:10,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33541.85 MB 2025-02-15 04:31:10,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41907.39 MB 2025-02-15 04:31:10,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 04:31:10,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29900.71 MB 2025-02-15 04:31:11,129 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 04:31:11,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:11,130 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:31:11,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:11,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:31:11,136 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:31:11,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:11,137 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:31:11,137 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:31:18,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:18,769 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:31:18,777 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:31:18,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:18,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:31:18,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:18,786 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:31:21,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:31:21,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:31:21,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.78 seconds 2025-02-15 04:31:21,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:21,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-15 04:31:21,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-15 04:31:21,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-15 04:31:21,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54454.65 MB 2025-02-15 04:31:21,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19295.90 MB 2025-02-15 04:31:21,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35158.75 MB 2025-02-15 04:31:21,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23652.54 MB 2025-02-15 04:31:21,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:31:21,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:31:21,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:31:21,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:21,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-15 04:31:21,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15003.98 MB 2025-02-15 04:31:21,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.04 MB 2025-02-15 04:31:21,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19295.90 MB 2025-02-15 04:31:21,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19295.90 MB 2025-02-15 04:31:21,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:21,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17058.42 MB 2025-02-15 04:31:22,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:31:22,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:31:22,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 04:31:22,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15003.98 MB 2025-02-15 04:31:22,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15217.65 MB 2025-02-15 04:31:22,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 04:31:22,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19295.90 MB 2025-02-15 04:31:22,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17658.02 MB 2025-02-15 04:31:22,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 04:31:22,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19174.67 MB 2025-02-15 04:31:22,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:31:22,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:31:22,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:31:22,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-15 04:31:22,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.94 MB 2025-02-15 04:31:22,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 04:31:22,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 04:31:22,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17658.02 MB 2025-02-15 04:31:22,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:22,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16548.46 MB 2025-02-15 04:31:22,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:31:22,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:31:22,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:31:22,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.94 MB 2025-02-15 04:31:22,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-15 04:31:22,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 04:31:22,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 04:31:22,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20329.79 MB 2025-02-15 04:31:22,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2671.77 MB 2025-02-15 04:31:22,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.69 MB 2025-02-15 04:31:22,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:31:22,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:31:22,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:31:22,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-15 04:31:22,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-15 04:31:22,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 04:31:22,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 04:31:22,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20329.79 MB 2025-02-15 04:31:22,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2671.77 MB 2025-02-15 04:31:22,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.69 MB 2025-02-15 04:31:22,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:31:22,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:31:22,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:31:22,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17497.57 MB 2025-02-15 04:31:22,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17808.13 MB 2025-02-15 04:31:22,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-15 04:31:22,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20329.79 MB 2025-02-15 04:31:22,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20495.47 MB 2025-02-15 04:31:22,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 04:31:22,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18102.35 MB 2025-02-15 04:31:22,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:31:22,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:31:22,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:31:22,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17974.32 MB 2025-02-15 04:31:22,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18201.95 MB 2025-02-15 04:31:22,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.62 MB 2025-02-15 04:31:22,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20495.47 MB 2025-02-15 04:31:22,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20495.47 MB 2025-02-15 04:31:22,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:22,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18224.57 MB 2025-02-15 04:31:22,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:31:22,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:31:22,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.76 seconds 2025-02-15 04:31:22,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-15 04:31:22,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18403.02 MB 2025-02-15 04:31:22,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4828.08 MB 2025-02-15 04:31:22,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54454.65 MB 2025-02-15 04:31:22,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20495.47 MB 2025-02-15 04:31:22,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33959.18 MB 2025-02-15 04:31:22,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18403.02 MB 2025-02-15 04:31:22,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:31:22,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:31:22,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:31:22,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.02 MB 2025-02-15 04:31:22,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17452.99 MB 2025-02-15 04:31:22,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -950.03 MB 2025-02-15 04:31:22,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20495.47 MB 2025-02-15 04:31:22,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20495.47 MB 2025-02-15 04:31:22,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:31:22,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19206.75 MB 2025-02-15 04:31:22,837 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:31:22,837 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-15 04:31:22,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:31:22,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:31:22,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:31:22,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:31:22,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17452.99 MB 2025-02-15 04:31:22,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25892.01 MB 2025-02-15 04:31:22,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:31:22,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20495.47 MB 2025-02-15 04:31:22,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-15 04:31:22,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:31:22,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.01 MB 2025-02-15 04:31:23,004 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:31:23,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:23,005 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:31:23,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:23,006 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:31:23,011 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:31:23,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:31:23,012 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:31:23,012 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-15 04:32:14,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:14,585 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:32:14,593 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:32:14,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:14,599 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:32:14,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:14,601 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:32:16,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:32:16,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:32:16,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.72 seconds 2025-02-15 04:32:16,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:16,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13728.24 MB 2025-02-15 04:32:16,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14113.98 MB 2025-02-15 04:32:16,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 385.74 MB 2025-02-15 04:32:16,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43570.43 MB 2025-02-15 04:32:16,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-15 04:32:16,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26002.59 MB 2025-02-15 04:32:16,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22973.11 MB 2025-02-15 04:32:16,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:32:16,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:32:16,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:32:16,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:16,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14113.98 MB 2025-02-15 04:32:16,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14300.87 MB 2025-02-15 04:32:16,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.89 MB 2025-02-15 04:32:16,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-15 04:32:16,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-15 04:32:16,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:16,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14879.55 MB 2025-02-15 04:32:16,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:32:16,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:32:16,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-15 04:32:16,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:16,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14300.87 MB 2025-02-15 04:32:16,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14445.53 MB 2025-02-15 04:32:16,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 144.65 MB 2025-02-15 04:32:16,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-15 04:32:16,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-15 04:32:16,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:16,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18386.63 MB 2025-02-15 04:32:16,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:32:16,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:32:16,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:32:16,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:16,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.46 MB 2025-02-15 04:32:16,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14960.23 MB 2025-02-15 04:32:16,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 514.77 MB 2025-02-15 04:32:16,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-15 04:32:16,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-15 04:32:16,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:16,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15346.49 MB 2025-02-15 04:32:17,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:32:17,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:32:17,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:32:17,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14960.23 MB 2025-02-15 04:32:17,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15585.47 MB 2025-02-15 04:32:17,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.24 MB 2025-02-15 04:32:17,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-15 04:32:17,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17825.79 MB 2025-02-15 04:32:17,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 257.95 MB 2025-02-15 04:32:17,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17082.74 MB 2025-02-15 04:32:17,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:32:17,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:32:17,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:32:17,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.46 MB 2025-02-15 04:32:17,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15585.47 MB 2025-02-15 04:32:17,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1140.01 MB 2025-02-15 04:32:17,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-15 04:32:17,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17825.79 MB 2025-02-15 04:32:17,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 257.95 MB 2025-02-15 04:32:17,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17082.74 MB 2025-02-15 04:32:17,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:32:17,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:32:17,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:32:17,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16189.09 MB 2025-02-15 04:32:17,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.68 MB 2025-02-15 04:32:17,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.58 MB 2025-02-15 04:32:17,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17825.79 MB 2025-02-15 04:32:17,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17993.56 MB 2025-02-15 04:32:17,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 04:32:17,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.55 MB 2025-02-15 04:32:17,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:32:17,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:32:17,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:32:17,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16617.77 MB 2025-02-15 04:32:17,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16846.32 MB 2025-02-15 04:32:17,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-15 04:32:17,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17993.56 MB 2025-02-15 04:32:17,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17993.56 MB 2025-02-15 04:32:17,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:17,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16846.32 MB 2025-02-15 04:32:17,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:32:17,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:32:17,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.52 seconds 2025-02-15 04:32:17,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13348.47 MB 2025-02-15 04:32:17,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.02 MB 2025-02-15 04:32:17,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3698.55 MB 2025-02-15 04:32:17,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43570.43 MB 2025-02-15 04:32:17,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17993.56 MB 2025-02-15 04:32:17,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25576.87 MB 2025-02-15 04:32:17,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17047.02 MB 2025-02-15 04:32:17,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:32:17,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:32:17,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 04:32:17,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17047.02 MB 2025-02-15 04:32:17,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20055.52 MB 2025-02-15 04:32:17,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-15 04:32:17,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17993.56 MB 2025-02-15 04:32:17,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21751.66 MB 2025-02-15 04:32:17,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3758.10 MB 2025-02-15 04:32:17,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20356.85 MB 2025-02-15 04:32:17,432 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 04:32:17,433 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2 ('] 2025-02-15 04:32:17,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:32:17,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:32:17,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:32:17,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:17,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20055.52 MB 2025-02-15 04:32:17,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28478.73 MB 2025-02-15 04:32:17,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 04:32:17,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21751.66 MB 2025-02-15 04:32:17,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32222.74 MB 2025-02-15 04:32:17,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 04:32:17,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28478.73 MB 2025-02-15 04:32:17,690 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 04:32:17,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:17,693 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:32:17,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:17,695 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:32:17,703 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:32:17,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:17,705 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:32:17,705 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2 ('] 2025-02-15 04:32:22,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:22,835 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:32:22,843 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:32:22,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:22,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:32:22,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:22,851 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:32:41,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:32:41,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:32:41,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.29 seconds 2025-02-15 04:32:41,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:41,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29376.38 MB 2025-02-15 04:32:41,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33562.95 MB 2025-02-15 04:32:41,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-15 04:32:41,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40600.86 MB 2025-02-15 04:32:41,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36840.67 MB 2025-02-15 04:32:41,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3760.19 MB 2025-02-15 04:32:41,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42471.63 MB 2025-02-15 04:32:41,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:32:41,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:32:41,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:32:41,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:41,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33562.95 MB 2025-02-15 04:32:41,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30092.24 MB 2025-02-15 04:32:41,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-15 04:32:41,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36840.67 MB 2025-02-15 04:32:41,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46468.69 MB 2025-02-15 04:32:41,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9628.02 MB 2025-02-15 04:32:41,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45998.80 MB 2025-02-15 04:32:43,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:32:43,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:32:43,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:32:43,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30092.24 MB 2025-02-15 04:32:43,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30623.08 MB 2025-02-15 04:32:43,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:32:43,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46468.69 MB 2025-02-15 04:32:43,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34068.23 MB 2025-02-15 04:32:43,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12400.46 MB 2025-02-15 04:32:43,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34602.66 MB 2025-02-15 04:32:43,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:32:43,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:32:43,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:32:43,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30623.08 MB 2025-02-15 04:32:43,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32512.28 MB 2025-02-15 04:32:43,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-15 04:32:43,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34068.23 MB 2025-02-15 04:32:43,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-15 04:32:43,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 04:32:43,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33929.71 MB 2025-02-15 04:32:43,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:32:43,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:32:43,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:32:43,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32512.28 MB 2025-02-15 04:32:43,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34755.19 MB 2025-02-15 04:32:43,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 04:32:43,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35957.77 MB 2025-02-15 04:32:43,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42331.01 MB 2025-02-15 04:32:43,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6373.24 MB 2025-02-15 04:32:43,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40299.11 MB 2025-02-15 04:32:43,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:32:43,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:32:43,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:32:43,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30623.08 MB 2025-02-15 04:32:43,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34755.19 MB 2025-02-15 04:32:43,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.11 MB 2025-02-15 04:32:43,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34068.23 MB 2025-02-15 04:32:43,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42331.01 MB 2025-02-15 04:32:43,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8262.78 MB 2025-02-15 04:32:43,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40299.11 MB 2025-02-15 04:32:43,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:32:43,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:32:43,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:32:43,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36288.73 MB 2025-02-15 04:32:43,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37055.73 MB 2025-02-15 04:32:43,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:32:43,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42331.01 MB 2025-02-15 04:32:43,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42748.35 MB 2025-02-15 04:32:43,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:32:43,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37763.52 MB 2025-02-15 04:32:43,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:32:43,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:32:43,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:32:43,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37468.62 MB 2025-02-15 04:32:43,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37696.78 MB 2025-02-15 04:32:43,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-15 04:32:43,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42748.35 MB 2025-02-15 04:32:43,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42748.35 MB 2025-02-15 04:32:43,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:43,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37935.59 MB 2025-02-15 04:32:43,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:32:43,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:32:43,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.74 seconds 2025-02-15 04:32:43,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25254.71 MB 2025-02-15 04:32:43,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37897.39 MB 2025-02-15 04:32:43,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12642.68 MB 2025-02-15 04:32:43,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40600.86 MB 2025-02-15 04:32:43,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42748.35 MB 2025-02-15 04:32:43,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2147.48 MB 2025-02-15 04:32:43,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37935.59 MB 2025-02-15 04:32:43,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:32:43,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:32:43,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:32:43,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37897.39 MB 2025-02-15 04:32:43,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30251.96 MB 2025-02-15 04:32:43,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7645.43 MB 2025-02-15 04:32:43,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42748.35 MB 2025-02-15 04:32:43,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42748.35 MB 2025-02-15 04:32:43,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:32:43,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40403.32 MB 2025-02-15 04:32:43,885 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 04:32:43,885 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:32:43,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:32:43,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:32:43,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:32:43,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:32:43,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30251.96 MB 2025-02-15 04:32:43,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38671.04 MB 2025-02-15 04:32:43,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 04:32:43,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42748.35 MB 2025-02-15 04:32:43,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46934.26 MB 2025-02-15 04:32:43,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 04:32:43,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38671.04 MB 2025-02-15 04:32:44,054 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 04:32:44,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:44,056 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:32:44,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:44,057 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:32:44,061 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:32:44,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:44,063 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:32:44,063 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:32:58,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:58,991 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:32:58,996 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:32:58,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:58,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 72, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:32:59,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:32:59,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 72, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:33:00,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:33:00,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:33:00,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.15 seconds 2025-02-15 04:33:00,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21634.75 MB 2025-02-15 04:33:00,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21889.56 MB 2025-02-15 04:33:00,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 254.80 MB 2025-02-15 04:33:00,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-15 04:33:00,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23362.27 MB 2025-02-15 04:33:00,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31943.82 MB 2025-02-15 04:33:00,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30879.63 MB 2025-02-15 04:33:00,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:33:00,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:33:00,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:33:00,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21889.56 MB 2025-02-15 04:33:00,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22013.01 MB 2025-02-15 04:33:00,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.45 MB 2025-02-15 04:33:00,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23362.27 MB 2025-02-15 04:33:00,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23362.27 MB 2025-02-15 04:33:00,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:00,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22395.28 MB 2025-02-15 04:33:00,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:33:00,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:33:00,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.36 seconds 2025-02-15 04:33:00,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22013.01 MB 2025-02-15 04:33:00,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22109.44 MB 2025-02-15 04:33:00,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 96.44 MB 2025-02-15 04:33:00,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23362.27 MB 2025-02-15 04:33:00,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23362.27 MB 2025-02-15 04:33:00,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:00,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26100.82 MB 2025-02-15 04:33:00,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:33:00,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:33:00,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:33:00,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22109.38 MB 2025-02-15 04:33:00,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22449.41 MB 2025-02-15 04:33:00,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.03 MB 2025-02-15 04:33:00,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23362.27 MB 2025-02-15 04:33:00,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23364.37 MB 2025-02-15 04:33:00,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 04:33:00,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22704.56 MB 2025-02-15 04:33:00,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:33:00,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:33:00,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:33:00,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22449.41 MB 2025-02-15 04:33:00,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22863.42 MB 2025-02-15 04:33:00,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 414.01 MB 2025-02-15 04:33:00,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23364.37 MB 2025-02-15 04:33:00,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24383.59 MB 2025-02-15 04:33:00,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1019.22 MB 2025-02-15 04:33:00,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23850.92 MB 2025-02-15 04:33:00,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:33:00,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:33:00,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:33:00,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22109.38 MB 2025-02-15 04:33:00,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22863.42 MB 2025-02-15 04:33:00,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 754.04 MB 2025-02-15 04:33:00,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23362.27 MB 2025-02-15 04:33:00,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24383.59 MB 2025-02-15 04:33:00,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1021.31 MB 2025-02-15 04:33:00,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23850.92 MB 2025-02-15 04:33:00,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:33:00,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:33:00,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:33:00,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23262.48 MB 2025-02-15 04:33:00,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23435.93 MB 2025-02-15 04:33:00,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 173.45 MB 2025-02-15 04:33:00,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24383.59 MB 2025-02-15 04:33:00,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24496.83 MB 2025-02-15 04:33:00,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 113.25 MB 2025-02-15 04:33:00,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23563.34 MB 2025-02-15 04:33:00,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:33:00,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:33:00,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:33:00,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23545.65 MB 2025-02-15 04:33:00,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23719.18 MB 2025-02-15 04:33:00,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 173.52 MB 2025-02-15 04:33:00,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24496.83 MB 2025-02-15 04:33:00,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24496.83 MB 2025-02-15 04:33:00,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:00,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23719.18 MB 2025-02-15 04:33:00,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:33:00,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:33:00,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.63 seconds 2025-02-15 04:33:00,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21383.90 MB 2025-02-15 04:33:00,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23873.74 MB 2025-02-15 04:33:00,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2489.84 MB 2025-02-15 04:33:00,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-15 04:33:00,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24496.83 MB 2025-02-15 04:33:00,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30809.26 MB 2025-02-15 04:33:00,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23873.74 MB 2025-02-15 04:33:00,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:33:00,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:33:00,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:33:00,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23873.74 MB 2025-02-15 04:33:00,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24119.84 MB 2025-02-15 04:33:00,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.10 MB 2025-02-15 04:33:00,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24496.83 MB 2025-02-15 04:33:00,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25440.55 MB 2025-02-15 04:33:00,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:33:00,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24351.51 MB 2025-02-15 04:33:00,874 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6271, cut from 6273 2025-02-15 04:33:00,874 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:33:00,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:33:00,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:33:00,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:33:00,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:00,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24119.84 MB 2025-02-15 04:33:00,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30606.66 MB 2025-02-15 04:33:00,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6486.82 MB 2025-02-15 04:33:00,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25440.55 MB 2025-02-15 04:33:00,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33504.10 MB 2025-02-15 04:33:00,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8063.55 MB 2025-02-15 04:33:00,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30606.66 MB 2025-02-15 04:33:01,001 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6063] 2025-02-15 04:33:01,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:01,002 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:33:01,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:01,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:33:01,008 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:33:01,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:01,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:33:01,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:33:11,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:11,446 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:33:11,454 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:33:11,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:11,460 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:33:11,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:11,462 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:33:15,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:33:15,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:33:15,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.07 seconds 2025-02-15 04:33:15,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:15,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22923.86 MB 2025-02-15 04:33:15,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23833.37 MB 2025-02-15 04:33:15,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-15 04:33:15,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39954.94 MB 2025-02-15 04:33:15,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26375.88 MB 2025-02-15 04:33:15,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13579.06 MB 2025-02-15 04:33:15,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32849.02 MB 2025-02-15 04:33:15,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:33:15,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:33:15,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:33:15,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:15,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23833.37 MB 2025-02-15 04:33:15,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24274.61 MB 2025-02-15 04:33:15,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 441.24 MB 2025-02-15 04:33:15,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26375.88 MB 2025-02-15 04:33:15,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29093.79 MB 2025-02-15 04:33:15,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2717.91 MB 2025-02-15 04:33:15,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27444.84 MB 2025-02-15 04:33:16,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:33:16,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:33:16,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.25 seconds 2025-02-15 04:33:16,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:16,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24274.61 MB 2025-02-15 04:33:16,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24615.68 MB 2025-02-15 04:33:16,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.07 MB 2025-02-15 04:33:16,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29093.79 MB 2025-02-15 04:33:16,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27728.54 MB 2025-02-15 04:33:16,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1365.25 MB 2025-02-15 04:33:16,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.17 MB 2025-02-15 04:33:16,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:33:16,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:33:16,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:33:16,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:16,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24615.68 MB 2025-02-15 04:33:16,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25829.50 MB 2025-02-15 04:33:16,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1213.82 MB 2025-02-15 04:33:16,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27728.54 MB 2025-02-15 04:33:16,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28946.99 MB 2025-02-15 04:33:16,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1218.45 MB 2025-02-15 04:33:16,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26740.20 MB 2025-02-15 04:33:16,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:33:16,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:33:16,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:33:16,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:16,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25829.50 MB 2025-02-15 04:33:16,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27269.91 MB 2025-02-15 04:33:16,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1440.42 MB 2025-02-15 04:33:16,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28946.99 MB 2025-02-15 04:33:16,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-15 04:33:16,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3649.04 MB 2025-02-15 04:33:16,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30833.93 MB 2025-02-15 04:33:16,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:33:16,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:33:16,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:33:16,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:16,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24615.68 MB 2025-02-15 04:33:16,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27269.91 MB 2025-02-15 04:33:16,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2654.23 MB 2025-02-15 04:33:16,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27728.54 MB 2025-02-15 04:33:16,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-15 04:33:16,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4867.49 MB 2025-02-15 04:33:16,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30833.93 MB 2025-02-15 04:33:17,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:33:17,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:33:17,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:33:17,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:17,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28255.21 MB 2025-02-15 04:33:17,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28749.85 MB 2025-02-15 04:33:17,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 494.63 MB 2025-02-15 04:33:17,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32596.03 MB 2025-02-15 04:33:17,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32864.47 MB 2025-02-15 04:33:17,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-15 04:33:17,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29204.60 MB 2025-02-15 04:33:17,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:33:17,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:33:17,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:33:17,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:17,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29015.13 MB 2025-02-15 04:33:17,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29245.76 MB 2025-02-15 04:33:17,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.62 MB 2025-02-15 04:33:17,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32864.47 MB 2025-02-15 04:33:17,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32864.47 MB 2025-02-15 04:33:17,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:17,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29340.12 MB 2025-02-15 04:33:17,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:33:17,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:33:17,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.60 seconds 2025-02-15 04:33:17,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:17,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22028.45 MB 2025-02-15 04:33:17,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29446.83 MB 2025-02-15 04:33:17,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7418.38 MB 2025-02-15 04:33:17,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39954.94 MB 2025-02-15 04:33:17,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32864.47 MB 2025-02-15 04:33:17,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7090.47 MB 2025-02-15 04:33:17,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29446.83 MB 2025-02-15 04:33:17,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:33:17,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:33:17,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:33:17,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:17,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29446.83 MB 2025-02-15 04:33:17,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32460.86 MB 2025-02-15 04:33:17,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:33:17,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32864.47 MB 2025-02-15 04:33:17,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33669.78 MB 2025-02-15 04:33:17,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 805.31 MB 2025-02-15 04:33:17,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32762.49 MB 2025-02-15 04:33:17,357 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:33:17,357 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:33:17,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:33:17,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:33:17,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:33:17,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:17,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26359.56 MB 2025-02-15 04:33:17,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34798.58 MB 2025-02-15 04:33:17,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:33:17,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33669.78 MB 2025-02-15 04:33:17,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44159.73 MB 2025-02-15 04:33:17,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:33:17,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34798.58 MB 2025-02-15 04:33:17,522 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:33:17,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:17,523 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:33:17,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:17,524 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:33:17,529 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:33:17,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:17,530 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:33:17,530 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:33:35,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:35,661 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:33:35,665 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:33:35,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:35,669 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:33:35,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:35,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:33:38,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:33:38,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:33:38,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.72 seconds 2025-02-15 04:33:38,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:38,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22352.47 MB 2025-02-15 04:33:38,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22971.79 MB 2025-02-15 04:33:38,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.32 MB 2025-02-15 04:33:38,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56744.74 MB 2025-02-15 04:33:38,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25478.30 MB 2025-02-15 04:33:38,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31266.44 MB 2025-02-15 04:33:38,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31824.10 MB 2025-02-15 04:33:38,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:33:38,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:33:38,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:33:38,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:38,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22971.79 MB 2025-02-15 04:33:38,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23272.56 MB 2025-02-15 04:33:38,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 300.78 MB 2025-02-15 04:33:38,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25478.30 MB 2025-02-15 04:33:38,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27332.18 MB 2025-02-15 04:33:38,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1853.88 MB 2025-02-15 04:33:38,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25430.63 MB 2025-02-15 04:33:39,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:33:39,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:33:39,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 04:33:39,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23272.56 MB 2025-02-15 04:33:39,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23504.81 MB 2025-02-15 04:33:39,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.24 MB 2025-02-15 04:33:39,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27332.18 MB 2025-02-15 04:33:39,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24465.38 MB 2025-02-15 04:33:39,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2866.81 MB 2025-02-15 04:33:39,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27446.32 MB 2025-02-15 04:33:39,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:33:39,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:33:39,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:33:39,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23504.74 MB 2025-02-15 04:33:39,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24331.74 MB 2025-02-15 04:33:39,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 826.99 MB 2025-02-15 04:33:39,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24465.38 MB 2025-02-15 04:33:39,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26122.13 MB 2025-02-15 04:33:39,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1656.75 MB 2025-02-15 04:33:39,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24952.13 MB 2025-02-15 04:33:39,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:33:39,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:33:39,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:33:39,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24331.74 MB 2025-02-15 04:33:39,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25313.99 MB 2025-02-15 04:33:39,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.26 MB 2025-02-15 04:33:39,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26122.13 MB 2025-02-15 04:33:39,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28808.58 MB 2025-02-15 04:33:39,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2686.45 MB 2025-02-15 04:33:39,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27740.63 MB 2025-02-15 04:33:39,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:33:39,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:33:39,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:33:39,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23504.74 MB 2025-02-15 04:33:39,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25313.99 MB 2025-02-15 04:33:39,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1809.25 MB 2025-02-15 04:33:39,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24465.38 MB 2025-02-15 04:33:39,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28808.58 MB 2025-02-15 04:33:39,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4343.20 MB 2025-02-15 04:33:39,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27740.63 MB 2025-02-15 04:33:39,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:33:39,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:33:39,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:33:39,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25984.92 MB 2025-02-15 04:33:39,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26320.48 MB 2025-02-15 04:33:39,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 335.56 MB 2025-02-15 04:33:39,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28808.58 MB 2025-02-15 04:33:39,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28991.03 MB 2025-02-15 04:33:39,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 04:33:39,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26638.32 MB 2025-02-15 04:33:39,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:33:39,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:33:39,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:33:39,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26501.13 MB 2025-02-15 04:33:39,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.45 MB 2025-02-15 04:33:39,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.32 MB 2025-02-15 04:33:39,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28991.03 MB 2025-02-15 04:33:39,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28991.03 MB 2025-02-15 04:33:39,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:39,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26753.58 MB 2025-02-15 04:33:39,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:33:39,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:33:39,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.78 seconds 2025-02-15 04:33:39,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21742.76 MB 2025-02-15 04:33:39,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26926.22 MB 2025-02-15 04:33:39,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5183.47 MB 2025-02-15 04:33:39,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56744.74 MB 2025-02-15 04:33:39,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28991.03 MB 2025-02-15 04:33:39,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27753.71 MB 2025-02-15 04:33:39,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26926.22 MB 2025-02-15 04:33:39,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:33:39,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:33:39,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:33:39,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26926.22 MB 2025-02-15 04:33:39,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25680.57 MB 2025-02-15 04:33:39,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1245.65 MB 2025-02-15 04:33:39,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28991.03 MB 2025-02-15 04:33:39,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28991.03 MB 2025-02-15 04:33:39,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:33:39,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27160.30 MB 2025-02-15 04:33:39,730 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 04:33:39,730 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 04:33:39,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:33:39,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:33:39,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:33:39,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:33:39,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25680.57 MB 2025-02-15 04:33:39,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34107.07 MB 2025-02-15 04:33:39,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 04:33:39,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28991.03 MB 2025-02-15 04:33:39,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39464.21 MB 2025-02-15 04:33:39,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 04:33:39,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34107.07 MB 2025-02-15 04:33:39,900 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 04:33:39,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:39,902 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:33:39,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:39,903 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:33:39,908 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:33:39,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:33:39,909 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:33:39,909 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 04:34:33,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:33,720 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:34:33,725 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:34:33,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:33,730 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 311, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:34:33,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:33,731 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 311, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:34:38,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:34:38,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:34:38,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.88 seconds 2025-02-15 04:34:38,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:38,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23300.14 MB 2025-02-15 04:34:38,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24401.15 MB 2025-02-15 04:34:38,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1101.00 MB 2025-02-15 04:34:38,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 04:34:38,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-15 04:34:38,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25423.77 MB 2025-02-15 04:34:38,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33225.30 MB 2025-02-15 04:34:38,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:34:38,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:34:38,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:34:38,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:38,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24401.15 MB 2025-02-15 04:34:38,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24926.91 MB 2025-02-15 04:34:38,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 525.76 MB 2025-02-15 04:34:38,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-15 04:34:38,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-15 04:34:38,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3812.62 MB 2025-02-15 04:34:38,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28756.34 MB 2025-02-15 04:34:40,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:34:40,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:34:40,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.51 seconds 2025-02-15 04:34:40,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24926.91 MB 2025-02-15 04:34:40,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25338.31 MB 2025-02-15 04:34:40,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 411.40 MB 2025-02-15 04:34:40,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-15 04:34:40,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26528.97 MB 2025-02-15 04:34:40,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3890.22 MB 2025-02-15 04:34:40,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29268.50 MB 2025-02-15 04:34:40,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:34:40,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:34:40,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:34:40,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.31 MB 2025-02-15 04:34:40,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26803.96 MB 2025-02-15 04:34:40,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1465.65 MB 2025-02-15 04:34:40,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26528.97 MB 2025-02-15 04:34:40,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29460.79 MB 2025-02-15 04:34:40,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2931.82 MB 2025-02-15 04:34:40,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27902.99 MB 2025-02-15 04:34:40,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:34:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:34:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:34:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26803.96 MB 2025-02-15 04:34:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28542.46 MB 2025-02-15 04:34:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1738.50 MB 2025-02-15 04:34:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29460.79 MB 2025-02-15 04:34:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34586.23 MB 2025-02-15 04:34:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5125.44 MB 2025-02-15 04:34:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32841.36 MB 2025-02-15 04:34:40,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:34:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:34:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 04:34:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.31 MB 2025-02-15 04:34:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28542.46 MB 2025-02-15 04:34:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3204.15 MB 2025-02-15 04:34:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26528.97 MB 2025-02-15 04:34:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34586.23 MB 2025-02-15 04:34:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8057.26 MB 2025-02-15 04:34:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32841.36 MB 2025-02-15 04:34:40,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:34:40,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:34:40,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:34:40,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29730.95 MB 2025-02-15 04:34:40,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30325.38 MB 2025-02-15 04:34:40,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.43 MB 2025-02-15 04:34:40,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34586.23 MB 2025-02-15 04:34:40,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 04:34:40,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 325.06 MB 2025-02-15 04:34:40,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30873.92 MB 2025-02-15 04:34:40,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:34:40,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:34:40,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:34:40,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30645.37 MB 2025-02-15 04:34:40,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30865.83 MB 2025-02-15 04:34:40,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.46 MB 2025-02-15 04:34:40,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 04:34:40,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 04:34:40,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:34:40,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31011.24 MB 2025-02-15 04:34:40,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:34:40,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:34:40,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.76 seconds 2025-02-15 04:34:40,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22216.59 MB 2025-02-15 04:34:40,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31066.90 MB 2025-02-15 04:34:40,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8850.31 MB 2025-02-15 04:34:40,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 04:34:40,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 04:34:40,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17119.05 MB 2025-02-15 04:34:40,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31066.90 MB 2025-02-15 04:34:40,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:34:40,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:34:40,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 04:34:40,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31066.90 MB 2025-02-15 04:34:40,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34080.94 MB 2025-02-15 04:34:40,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:34:40,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 04:34:40,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35313.94 MB 2025-02-15 04:34:40,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 04:34:40,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34382.57 MB 2025-02-15 04:34:40,776 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:34:40,776 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 04:34:40,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:34:40,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:34:40,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:34:40,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:34:40,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26796.25 MB 2025-02-15 04:34:40,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35235.27 MB 2025-02-15 04:34:40,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:34:40,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35313.94 MB 2025-02-15 04:34:40,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45803.90 MB 2025-02-15 04:34:40,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:34:40,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35235.27 MB 2025-02-15 04:34:40,946 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:34:40,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:40,947 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:34:40,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:40,948 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:34:40,953 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:34:40,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:34:40,954 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:34:40,954 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 04:35:34,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:34,921 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:35:34,926 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:35:34,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:34,931 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:35:34,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:34,932 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:35:53,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:35:53,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:35:53,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.73 seconds 2025-02-15 04:35:53,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:53,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29613.29 MB 2025-02-15 04:35:53,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33920.84 MB 2025-02-15 04:35:53,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 04:35:53,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58388.91 MB 2025-02-15 04:35:53,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39072.04 MB 2025-02-15 04:35:53,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19316.87 MB 2025-02-15 04:35:53,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42935.04 MB 2025-02-15 04:35:53,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:35:53,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:35:53,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:35:53,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:53,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33920.84 MB 2025-02-15 04:35:53,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30270.04 MB 2025-02-15 04:35:53,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3650.80 MB 2025-02-15 04:35:53,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39072.04 MB 2025-02-15 04:35:53,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49746.54 MB 2025-02-15 04:35:53,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10674.50 MB 2025-02-15 04:35:53,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46776.38 MB 2025-02-15 04:35:55,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:35:55,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:35:55,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:35:55,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:55,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30270.04 MB 2025-02-15 04:35:55,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30800.88 MB 2025-02-15 04:35:55,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:35:55,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49746.54 MB 2025-02-15 04:35:55,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32694.60 MB 2025-02-15 04:35:55,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17051.94 MB 2025-02-15 04:35:55,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.47 MB 2025-02-15 04:35:55,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:35:55,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:35:55,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:35:55,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:55,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30800.88 MB 2025-02-15 04:35:55,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32690.42 MB 2025-02-15 04:35:55,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:35:55,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32694.60 MB 2025-02-15 04:35:55,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35999.71 MB 2025-02-15 04:35:55,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 04:35:55,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34107.84 MB 2025-02-15 04:35:55,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:35:55,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:35:55,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:35:55,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:55,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32690.42 MB 2025-02-15 04:35:55,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34932.27 MB 2025-02-15 04:35:55,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:35:55,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35999.71 MB 2025-02-15 04:35:55,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42607.84 MB 2025-02-15 04:35:55,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 04:35:55,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40477.24 MB 2025-02-15 04:35:55,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:35:55,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:35:55,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:35:55,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:55,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30800.88 MB 2025-02-15 04:35:55,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34932.27 MB 2025-02-15 04:35:55,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:35:55,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32694.60 MB 2025-02-15 04:35:55,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42607.84 MB 2025-02-15 04:35:55,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9913.24 MB 2025-02-15 04:35:55,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40477.24 MB 2025-02-15 04:35:56,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:35:56,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:35:56,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:35:56,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:56,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36466.50 MB 2025-02-15 04:35:56,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37233.50 MB 2025-02-15 04:35:56,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:35:56,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42607.84 MB 2025-02-15 04:35:56,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-15 04:35:56,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:35:56,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37941.29 MB 2025-02-15 04:35:56,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:35:56,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:35:56,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:35:56,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:56,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37646.39 MB 2025-02-15 04:35:56,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37875.43 MB 2025-02-15 04:35:56,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-15 04:35:56,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-15 04:35:56,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-15 04:35:56,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:35:56,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38119.70 MB 2025-02-15 04:35:56,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:35:56,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:35:56,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.23 seconds 2025-02-15 04:35:56,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:56,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25373.17 MB 2025-02-15 04:35:56,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38076.38 MB 2025-02-15 04:35:56,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12703.21 MB 2025-02-15 04:35:56,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58388.91 MB 2025-02-15 04:35:56,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-15 04:35:56,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15363.74 MB 2025-02-15 04:35:56,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38119.70 MB 2025-02-15 04:35:56,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:35:56,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:35:56,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:35:56,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:56,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38076.38 MB 2025-02-15 04:35:56,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30376.34 MB 2025-02-15 04:35:56,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7700.04 MB 2025-02-15 04:35:56,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-15 04:35:56,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-15 04:35:56,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:35:56,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40586.51 MB 2025-02-15 04:35:56,456 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 04:35:56,456 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:35:56,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:35:56,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:35:56,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:35:56,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:35:56,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30376.34 MB 2025-02-15 04:35:56,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38810.96 MB 2025-02-15 04:35:56,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 04:35:56,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-15 04:35:56,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53506.74 MB 2025-02-15 04:35:56,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 04:35:56,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38810.96 MB 2025-02-15 04:35:56,624 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 04:35:56,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:56,625 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:35:56,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:56,626 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:35:56,631 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:35:56,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:35:56,632 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:35:56,632 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:37:03,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:03,191 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:37:03,196 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:37:03,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:03,200 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1306, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:37:03,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:03,201 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1306, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:37:23,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:37:23,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:37:23,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.02 seconds 2025-02-15 04:37:23,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:23,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30233.46 MB 2025-02-15 04:37:23,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34855.58 MB 2025-02-15 04:37:23,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4622.12 MB 2025-02-15 04:37:23,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61891.15 MB 2025-02-15 04:37:23,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43572.53 MB 2025-02-15 04:37:23,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18318.62 MB 2025-02-15 04:37:23,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43781.69 MB 2025-02-15 04:37:23,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:37:23,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:37:23,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:37:23,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:23,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34855.58 MB 2025-02-15 04:37:23,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30731.67 MB 2025-02-15 04:37:23,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4123.91 MB 2025-02-15 04:37:23,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43572.53 MB 2025-02-15 04:37:23,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52745.47 MB 2025-02-15 04:37:23,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9172.94 MB 2025-02-15 04:37:23,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48650.69 MB 2025-02-15 04:37:25,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:37:25,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:37:25,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:37:25,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30731.67 MB 2025-02-15 04:37:25,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31262.52 MB 2025-02-15 04:37:25,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:37:25,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52745.47 MB 2025-02-15 04:37:25,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34758.20 MB 2025-02-15 04:37:25,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17987.27 MB 2025-02-15 04:37:25,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35242.10 MB 2025-02-15 04:37:25,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:37:25,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:37:25,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:37:25,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31262.52 MB 2025-02-15 04:37:25,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33152.05 MB 2025-02-15 04:37:25,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:37:25,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34758.20 MB 2025-02-15 04:37:25,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36647.73 MB 2025-02-15 04:37:25,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 04:37:25,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34569.48 MB 2025-02-15 04:37:25,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:37:25,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:37:25,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:37:25,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33152.05 MB 2025-02-15 04:37:25,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35394.95 MB 2025-02-15 04:37:25,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 04:37:25,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36647.73 MB 2025-02-15 04:37:25,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43020.98 MB 2025-02-15 04:37:25,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6373.24 MB 2025-02-15 04:37:25,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40938.88 MB 2025-02-15 04:37:25,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:37:25,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:37:25,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:37:25,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31262.52 MB 2025-02-15 04:37:25,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35394.95 MB 2025-02-15 04:37:25,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 04:37:25,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34758.20 MB 2025-02-15 04:37:25,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43020.98 MB 2025-02-15 04:37:25,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8262.78 MB 2025-02-15 04:37:25,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40938.88 MB 2025-02-15 04:37:25,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:37:25,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:37:25,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:37:25,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36928.50 MB 2025-02-15 04:37:25,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37695.50 MB 2025-02-15 04:37:25,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:37:25,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43020.98 MB 2025-02-15 04:37:25,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43438.31 MB 2025-02-15 04:37:25,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:37:25,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38403.29 MB 2025-02-15 04:37:25,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:37:25,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:37:25,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:37:25,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38108.39 MB 2025-02-15 04:37:25,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38336.96 MB 2025-02-15 04:37:25,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-15 04:37:25,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43438.31 MB 2025-02-15 04:37:25,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43438.31 MB 2025-02-15 04:37:25,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:37:25,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38572.82 MB 2025-02-15 04:37:25,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:37:25,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:37:25,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.46 seconds 2025-02-15 04:37:25,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25683.25 MB 2025-02-15 04:37:25,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38537.66 MB 2025-02-15 04:37:25,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12854.41 MB 2025-02-15 04:37:25,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61891.15 MB 2025-02-15 04:37:25,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43438.31 MB 2025-02-15 04:37:25,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18452.84 MB 2025-02-15 04:37:25,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38572.82 MB 2025-02-15 04:37:25,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:37:25,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:37:25,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:37:25,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38537.66 MB 2025-02-15 04:37:25,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30679.63 MB 2025-02-15 04:37:25,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7858.03 MB 2025-02-15 04:37:25,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43438.31 MB 2025-02-15 04:37:25,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43438.31 MB 2025-02-15 04:37:25,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:37:25,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41042.86 MB 2025-02-15 04:37:25,950 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 04:37:25,950 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:37:25,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:37:25,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:37:25,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:37:25,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:37:25,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30679.63 MB 2025-02-15 04:37:25,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39093.54 MB 2025-02-15 04:37:25,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.92 MB 2025-02-15 04:37:25,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43438.31 MB 2025-02-15 04:37:25,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47622.13 MB 2025-02-15 04:37:25,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 04:37:25,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39093.54 MB 2025-02-15 04:37:26,115 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 04:37:26,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:26,116 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:37:26,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:26,117 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:37:26,122 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:37:26,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:37:26,123 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:37:26,123 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:38:29,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:29,752 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:38:29,757 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:38:29,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:29,761 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1440, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:38:29,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:29,762 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1440, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:38:51,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:38:51,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:38:51,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.15 seconds 2025-02-15 04:38:51,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:51,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31167.19 MB 2025-02-15 04:38:51,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36263.27 MB 2025-02-15 04:38:51,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5096.08 MB 2025-02-15 04:38:51,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55985.57 MB 2025-02-15 04:38:51,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44017.12 MB 2025-02-15 04:38:51,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11968.45 MB 2025-02-15 04:38:51,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45168.41 MB 2025-02-15 04:38:52,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:38:52,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:38:52,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:38:52,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:52,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36263.27 MB 2025-02-15 04:38:52,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31428.30 MB 2025-02-15 04:38:52,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4834.97 MB 2025-02-15 04:38:52,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44017.12 MB 2025-02-15 04:38:52,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53670.31 MB 2025-02-15 04:38:52,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9653.19 MB 2025-02-15 04:38:52,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50777.01 MB 2025-02-15 04:38:53,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:38:53,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:38:53,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:38:53,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:53,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31428.30 MB 2025-02-15 04:38:53,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31959.14 MB 2025-02-15 04:38:53,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:38:53,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53670.31 MB 2025-02-15 04:38:53,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34737.23 MB 2025-02-15 04:38:53,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18933.09 MB 2025-02-15 04:38:53,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35937.69 MB 2025-02-15 04:38:53,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:38:53,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:38:53,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:38:53,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:53,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31959.14 MB 2025-02-15 04:38:53,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33848.67 MB 2025-02-15 04:38:53,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:38:53,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34737.23 MB 2025-02-15 04:38:53,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37570.48 MB 2025-02-15 04:38:53,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 04:38:53,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35266.10 MB 2025-02-15 04:38:54,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:38:54,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:38:54,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:38:54,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33848.67 MB 2025-02-15 04:38:54,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36090.53 MB 2025-02-15 04:38:54,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:38:54,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37570.48 MB 2025-02-15 04:38:54,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43706.74 MB 2025-02-15 04:38:54,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 04:38:54,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41635.50 MB 2025-02-15 04:38:54,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:38:54,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:38:54,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:38:54,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31959.14 MB 2025-02-15 04:38:54,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36090.53 MB 2025-02-15 04:38:54,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:38:54,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34737.23 MB 2025-02-15 04:38:54,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43706.74 MB 2025-02-15 04:38:54,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8969.52 MB 2025-02-15 04:38:54,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41635.50 MB 2025-02-15 04:38:54,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:38:54,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:38:54,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:38:54,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37624.76 MB 2025-02-15 04:38:54,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38391.76 MB 2025-02-15 04:38:54,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:38:54,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43706.74 MB 2025-02-15 04:38:54,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44124.08 MB 2025-02-15 04:38:54,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:38:54,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39099.55 MB 2025-02-15 04:38:54,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:38:54,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:38:54,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:38:54,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38804.65 MB 2025-02-15 04:38:54,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39033.61 MB 2025-02-15 04:38:54,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 04:38:54,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44124.08 MB 2025-02-15 04:38:54,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44124.08 MB 2025-02-15 04:38:54,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:38:54,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39250.43 MB 2025-02-15 04:38:54,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:38:54,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:38:54,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.58 seconds 2025-02-15 04:38:54,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26150.12 MB 2025-02-15 04:38:54,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39234.49 MB 2025-02-15 04:38:54,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13084.37 MB 2025-02-15 04:38:54,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55985.57 MB 2025-02-15 04:38:54,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44124.08 MB 2025-02-15 04:38:54,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11861.49 MB 2025-02-15 04:38:54,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39250.43 MB 2025-02-15 04:38:54,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:38:54,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:38:54,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:38:54,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39234.49 MB 2025-02-15 04:38:54,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31152.15 MB 2025-02-15 04:38:54,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8082.34 MB 2025-02-15 04:38:54,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44124.08 MB 2025-02-15 04:38:54,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44124.08 MB 2025-02-15 04:38:54,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:38:54,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41743.70 MB 2025-02-15 04:38:54,633 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 04:38:54,634 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:38:54,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:38:54,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:38:54,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:38:54,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:38:54,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31152.15 MB 2025-02-15 04:38:54,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39582.82 MB 2025-02-15 04:38:54,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 04:38:54,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44124.08 MB 2025-02-15 04:38:54,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52506.39 MB 2025-02-15 04:38:54,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 04:38:54,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39582.82 MB 2025-02-15 04:38:54,798 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 04:38:54,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:54,800 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:38:54,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:54,801 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:38:54,805 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:38:54,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:38:54,806 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:38:54,806 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:39:33,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:39:33,644 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:39:33,652 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:39:33,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:39:33,659 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1587, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:39:33,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:39:33,661 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1587, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:39:58,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:39:58,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:39:58,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.66 seconds 2025-02-15 04:39:58,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:39:58,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32191.51 MB 2025-02-15 04:39:58,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37807.82 MB 2025-02-15 04:39:58,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5616.30 MB 2025-02-15 04:39:58,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65078.82 MB 2025-02-15 04:39:58,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44545.61 MB 2025-02-15 04:39:58,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20533.22 MB 2025-02-15 04:39:58,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46646.53 MB 2025-02-15 04:39:58,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:39:58,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:39:58,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 04:39:58,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:39:58,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37807.82 MB 2025-02-15 04:39:58,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32192.51 MB 2025-02-15 04:39:58,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5615.31 MB 2025-02-15 04:39:58,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44545.61 MB 2025-02-15 04:39:58,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56430.17 MB 2025-02-15 04:39:58,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11884.56 MB 2025-02-15 04:39:58,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53548.01 MB 2025-02-15 04:40:00,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:40:00,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:40:00,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:40:00,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32192.51 MB 2025-02-15 04:40:00,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32723.35 MB 2025-02-15 04:40:00,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:40:00,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56430.17 MB 2025-02-15 04:40:00,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36161.19 MB 2025-02-15 04:40:00,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20268.97 MB 2025-02-15 04:40:00,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36702.93 MB 2025-02-15 04:40:00,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:40:00,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:40:00,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:40:00,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32723.35 MB 2025-02-15 04:40:00,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34612.88 MB 2025-02-15 04:40:00,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:40:00,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36161.19 MB 2025-02-15 04:40:00,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38050.73 MB 2025-02-15 04:40:00,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 04:40:00,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36030.31 MB 2025-02-15 04:40:00,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:40:00,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:40:00,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:40:00,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34612.88 MB 2025-02-15 04:40:00,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36855.43 MB 2025-02-15 04:40:00,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.54 MB 2025-02-15 04:40:00,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38050.73 MB 2025-02-15 04:40:00,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44186.99 MB 2025-02-15 04:40:00,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 04:40:00,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42399.71 MB 2025-02-15 04:40:00,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:40:00,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:40:00,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:40:00,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32723.35 MB 2025-02-15 04:40:00,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36855.43 MB 2025-02-15 04:40:00,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.08 MB 2025-02-15 04:40:00,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36161.19 MB 2025-02-15 04:40:00,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44186.99 MB 2025-02-15 04:40:00,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8025.80 MB 2025-02-15 04:40:00,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42399.71 MB 2025-02-15 04:40:00,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:40:00,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:40:00,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:40:00,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38388.97 MB 2025-02-15 04:40:00,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39155.97 MB 2025-02-15 04:40:00,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:40:00,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44186.99 MB 2025-02-15 04:40:00,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44604.33 MB 2025-02-15 04:40:00,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:40:00,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39863.76 MB 2025-02-15 04:40:00,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:40:00,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:40:00,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:40:00,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39568.86 MB 2025-02-15 04:40:00,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39797.82 MB 2025-02-15 04:40:00,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 04:40:00,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44604.33 MB 2025-02-15 04:40:00,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44604.33 MB 2025-02-15 04:40:00,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:40:00,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40004.75 MB 2025-02-15 04:40:00,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:40:00,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:40:00,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.16 seconds 2025-02-15 04:40:00,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:00,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26662.28 MB 2025-02-15 04:40:00,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39998.70 MB 2025-02-15 04:40:00,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13336.42 MB 2025-02-15 04:40:00,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65078.82 MB 2025-02-15 04:40:00,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44604.33 MB 2025-02-15 04:40:00,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20474.49 MB 2025-02-15 04:40:00,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40004.75 MB 2025-02-15 04:40:01,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:40:01,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:40:01,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:40:01,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:01,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39998.70 MB 2025-02-15 04:40:01,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31663.62 MB 2025-02-15 04:40:01,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8335.08 MB 2025-02-15 04:40:01,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44604.33 MB 2025-02-15 04:40:01,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44604.33 MB 2025-02-15 04:40:01,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:40:01,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42507.91 MB 2025-02-15 04:40:01,113 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 04:40:01,113 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:40:01,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:40:01,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:40:01,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:40:01,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:01,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31663.62 MB 2025-02-15 04:40:01,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40094.30 MB 2025-02-15 04:40:01,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 04:40:01,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44604.33 MB 2025-02-15 04:40:01,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52986.64 MB 2025-02-15 04:40:01,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 04:40:01,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40094.30 MB 2025-02-15 04:40:01,277 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 04:40:01,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:01,278 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:40:01,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:01,279 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:40:01,284 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:40:01,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:01,285 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:40:01,285 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:40:43,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:43,664 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:40:43,669 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:40:43,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:43,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 876, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:40:43,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:40:43,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 876, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:40:57,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:40:57,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:40:57,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.60 seconds 2025-02-15 04:40:57,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:57,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27237.15 MB 2025-02-15 04:40:57,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30337.27 MB 2025-02-15 04:40:57,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3100.11 MB 2025-02-15 04:40:57,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65559.07 MB 2025-02-15 04:40:57,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33655.10 MB 2025-02-15 04:40:57,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31903.97 MB 2025-02-15 04:40:57,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39200.75 MB 2025-02-15 04:40:57,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:40:57,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:40:57,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:40:57,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:57,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30337.27 MB 2025-02-15 04:40:57,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28497.29 MB 2025-02-15 04:40:57,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1839.98 MB 2025-02-15 04:40:57,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33655.10 MB 2025-02-15 04:40:57,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41232.11 MB 2025-02-15 04:40:57,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7577.01 MB 2025-02-15 04:40:57,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40014.54 MB 2025-02-15 04:40:59,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:40:59,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:40:59,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:40:59,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28497.29 MB 2025-02-15 04:40:59,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29028.13 MB 2025-02-15 04:40:59,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:40:59,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41232.11 MB 2025-02-15 04:40:59,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32677.82 MB 2025-02-15 04:40:59,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8554.28 MB 2025-02-15 04:40:59,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33007.72 MB 2025-02-15 04:40:59,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:40:59,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:40:59,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:40:59,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29028.13 MB 2025-02-15 04:40:59,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30917.66 MB 2025-02-15 04:40:59,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:40:59,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32677.82 MB 2025-02-15 04:40:59,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-15 04:40:59,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 04:40:59,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32335.09 MB 2025-02-15 04:40:59,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:40:59,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:40:59,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:40:59,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30917.66 MB 2025-02-15 04:40:59,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33160.21 MB 2025-02-15 04:40:59,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.54 MB 2025-02-15 04:40:59,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34567.36 MB 2025-02-15 04:40:59,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41175.48 MB 2025-02-15 04:40:59,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 04:40:59,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38704.49 MB 2025-02-15 04:40:59,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:40:59,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:40:59,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:40:59,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29028.13 MB 2025-02-15 04:40:59,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33160.21 MB 2025-02-15 04:40:59,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.08 MB 2025-02-15 04:40:59,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32677.82 MB 2025-02-15 04:40:59,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41175.48 MB 2025-02-15 04:40:59,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8497.66 MB 2025-02-15 04:40:59,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38704.49 MB 2025-02-15 04:40:59,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:40:59,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:40:59,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:40:59,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34693.75 MB 2025-02-15 04:40:59,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35460.75 MB 2025-02-15 04:40:59,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:40:59,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41175.48 MB 2025-02-15 04:40:59,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41592.82 MB 2025-02-15 04:40:59,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:40:59,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36168.54 MB 2025-02-15 04:40:59,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:40:59,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:40:59,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:40:59,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35873.64 MB 2025-02-15 04:40:59,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36101.82 MB 2025-02-15 04:40:59,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 04:40:59,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41592.82 MB 2025-02-15 04:40:59,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41592.82 MB 2025-02-15 04:40:59,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:40:59,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36297.85 MB 2025-02-15 04:40:59,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:40:59,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:40:59,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.02 seconds 2025-02-15 04:40:59,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24185.10 MB 2025-02-15 04:40:59,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36302.52 MB 2025-02-15 04:40:59,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12117.42 MB 2025-02-15 04:40:59,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65559.07 MB 2025-02-15 04:40:59,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41592.82 MB 2025-02-15 04:40:59,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23966.25 MB 2025-02-15 04:40:59,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36302.52 MB 2025-02-15 04:40:59,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:40:59,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:40:59,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:40:59,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36302.52 MB 2025-02-15 04:40:59,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29180.57 MB 2025-02-15 04:40:59,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7121.95 MB 2025-02-15 04:40:59,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41592.82 MB 2025-02-15 04:40:59,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41592.82 MB 2025-02-15 04:40:59,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:40:59,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38806.81 MB 2025-02-15 04:40:59,981 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 04:40:59,981 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:40:59,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:40:59,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:40:59,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:40:59,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:40:59,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29180.57 MB 2025-02-15 04:40:59,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37594.55 MB 2025-02-15 04:40:59,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 04:40:59,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41592.82 MB 2025-02-15 04:40:59,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49958.35 MB 2025-02-15 04:40:59,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 04:40:59,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37594.55 MB 2025-02-15 04:41:00,145 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 04:41:00,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:00,147 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:41:00,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:00,148 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:41:00,152 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:41:00,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:00,154 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:41:00,154 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:41:10,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:10,383 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:41:10,388 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:41:10,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:10,391 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:41:10,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:10,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:41:23,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:41:23,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:41:23,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.61 seconds 2025-02-15 04:41:23,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:23,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26763.32 MB 2025-02-15 04:41:23,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29623.83 MB 2025-02-15 04:41:23,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-15 04:41:23,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 04:41:23,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34829.50 MB 2025-02-15 04:41:23,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27676.11 MB 2025-02-15 04:41:23,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38499.61 MB 2025-02-15 04:41:23,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:41:23,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:41:23,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:41:23,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:23,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29623.83 MB 2025-02-15 04:41:23,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28143.78 MB 2025-02-15 04:41:23,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1480.06 MB 2025-02-15 04:41:23,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34829.50 MB 2025-02-15 04:41:23,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42364.57 MB 2025-02-15 04:41:23,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7535.07 MB 2025-02-15 04:41:23,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39259.01 MB 2025-02-15 04:41:25,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:41:25,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:41:25,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 04:41:25,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28143.78 MB 2025-02-15 04:41:25,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28674.62 MB 2025-02-15 04:41:25,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:41:25,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42364.57 MB 2025-02-15 04:41:25,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29911.68 MB 2025-02-15 04:41:25,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12452.89 MB 2025-02-15 04:41:25,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32654.20 MB 2025-02-15 04:41:25,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:41:25,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:41:25,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:41:25,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28674.62 MB 2025-02-15 04:41:25,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30564.15 MB 2025-02-15 04:41:25,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:41:25,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29911.68 MB 2025-02-15 04:41:25,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-15 04:41:25,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-15 04:41:25,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31981.58 MB 2025-02-15 04:41:25,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:41:25,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:41:25,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:41:25,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30564.15 MB 2025-02-15 04:41:25,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32806.70 MB 2025-02-15 04:41:25,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.54 MB 2025-02-15 04:41:25,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-15 04:41:25,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39824.92 MB 2025-02-15 04:41:25,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 04:41:25,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38350.98 MB 2025-02-15 04:41:25,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:41:25,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:41:25,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:41:25,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28674.62 MB 2025-02-15 04:41:25,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32806.70 MB 2025-02-15 04:41:25,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.08 MB 2025-02-15 04:41:25,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29911.68 MB 2025-02-15 04:41:25,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39824.92 MB 2025-02-15 04:41:25,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9913.24 MB 2025-02-15 04:41:25,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38350.98 MB 2025-02-15 04:41:25,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:41:25,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:41:25,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 04:41:25,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34340.24 MB 2025-02-15 04:41:25,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35107.24 MB 2025-02-15 04:41:25,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:41:25,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39824.92 MB 2025-02-15 04:41:25,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40242.25 MB 2025-02-15 04:41:25,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 04:41:25,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35815.03 MB 2025-02-15 04:41:25,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:41:25,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:41:25,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:41:25,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35520.13 MB 2025-02-15 04:41:25,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35748.75 MB 2025-02-15 04:41:25,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-15 04:41:25,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40242.25 MB 2025-02-15 04:41:25,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40242.25 MB 2025-02-15 04:41:25,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:25,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35958.44 MB 2025-02-15 04:41:25,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:41:25,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:41:25,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.07 seconds 2025-02-15 04:41:25,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23948.18 MB 2025-02-15 04:41:25,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35949.45 MB 2025-02-15 04:41:25,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12001.27 MB 2025-02-15 04:41:25,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 04:41:25,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40242.25 MB 2025-02-15 04:41:25,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22263.37 MB 2025-02-15 04:41:25,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35958.44 MB 2025-02-15 04:41:25,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:41:25,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:41:25,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 04:41:25,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35949.45 MB 2025-02-15 04:41:25,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28944.36 MB 2025-02-15 04:41:25,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7005.09 MB 2025-02-15 04:41:25,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40242.25 MB 2025-02-15 04:41:25,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40242.25 MB 2025-02-15 04:41:25,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:25,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38454.36 MB 2025-02-15 04:41:25,772 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 04:41:25,772 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:41:25,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:41:25,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:41:25,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:41:25,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:25,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28944.36 MB 2025-02-15 04:41:25,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37360.96 MB 2025-02-15 04:41:25,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 04:41:25,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40242.25 MB 2025-02-15 04:41:25,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50702.84 MB 2025-02-15 04:41:25,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-15 04:41:25,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37360.96 MB 2025-02-15 04:41:26,016 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 04:41:26,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:26,017 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:41:26,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:26,018 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:41:26,024 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:41:26,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:26,025 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:41:26,025 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 04:41:31,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:31,646 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:41:31,651 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:41:31,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:31,654 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:41:31,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:31,655 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:41:34,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:41:34,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:41:34,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.74 seconds 2025-02-15 04:41:34,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:34,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22345.50 MB 2025-02-15 04:41:34,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22962.07 MB 2025-02-15 04:41:34,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 616.56 MB 2025-02-15 04:41:34,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59070.48 MB 2025-02-15 04:41:34,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25157.44 MB 2025-02-15 04:41:34,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33913.04 MB 2025-02-15 04:41:34,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31817.68 MB 2025-02-15 04:41:34,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:41:34,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:41:34,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:41:34,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:34,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22962.07 MB 2025-02-15 04:41:34,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23170.08 MB 2025-02-15 04:41:34,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.01 MB 2025-02-15 04:41:34,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25157.44 MB 2025-02-15 04:41:34,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27137.15 MB 2025-02-15 04:41:34,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1979.71 MB 2025-02-15 04:41:34,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25226.73 MB 2025-02-15 04:41:35,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:41:35,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:41:35,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 04:41:35,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23170.08 MB 2025-02-15 04:41:35,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23383.74 MB 2025-02-15 04:41:35,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 04:41:35,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27137.15 MB 2025-02-15 04:41:35,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25201.48 MB 2025-02-15 04:41:35,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1935.67 MB 2025-02-15 04:41:35,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27341.80 MB 2025-02-15 04:41:35,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:41:35,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:41:35,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:41:35,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23383.68 MB 2025-02-15 04:41:35,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24144.03 MB 2025-02-15 04:41:35,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 04:41:35,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25201.48 MB 2025-02-15 04:41:35,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25966.94 MB 2025-02-15 04:41:35,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 765.46 MB 2025-02-15 04:41:35,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24714.55 MB 2025-02-15 04:41:35,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:41:35,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:41:35,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:41:35,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24144.03 MB 2025-02-15 04:41:35,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25046.41 MB 2025-02-15 04:41:35,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 04:41:35,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25966.94 MB 2025-02-15 04:41:35,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28638.71 MB 2025-02-15 04:41:35,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2671.77 MB 2025-02-15 04:41:35,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27279.78 MB 2025-02-15 04:41:35,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:41:35,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:41:35,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:41:35,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23383.68 MB 2025-02-15 04:41:35,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25046.41 MB 2025-02-15 04:41:35,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 04:41:35,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25201.48 MB 2025-02-15 04:41:35,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28638.71 MB 2025-02-15 04:41:35,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3437.23 MB 2025-02-15 04:41:35,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27279.78 MB 2025-02-15 04:41:35,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:41:35,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:41:35,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:41:35,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25663.67 MB 2025-02-15 04:41:35,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25974.22 MB 2025-02-15 04:41:35,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-15 04:41:35,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28638.71 MB 2025-02-15 04:41:35,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-15 04:41:35,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 04:41:35,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26269.06 MB 2025-02-15 04:41:35,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:41:35,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:41:35,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:41:35,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26140.41 MB 2025-02-15 04:41:35,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26367.67 MB 2025-02-15 04:41:35,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.25 MB 2025-02-15 04:41:35,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-15 04:41:35,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-15 04:41:35,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:35,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26386.80 MB 2025-02-15 04:41:35,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:41:35,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:41:35,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-15 04:41:35,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21739.27 MB 2025-02-15 04:41:35,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26568.74 MB 2025-02-15 04:41:35,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4829.46 MB 2025-02-15 04:41:35,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59070.48 MB 2025-02-15 04:41:35,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-15 04:41:35,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30264.00 MB 2025-02-15 04:41:35,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26568.74 MB 2025-02-15 04:41:35,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:41:35,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:41:35,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:41:35,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26568.74 MB 2025-02-15 04:41:35,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25616.96 MB 2025-02-15 04:41:35,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -951.77 MB 2025-02-15 04:41:35,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-15 04:41:35,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-15 04:41:35,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:35,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27372.47 MB 2025-02-15 04:41:35,660 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:41:35,661 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-15 04:41:35,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:41:35,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:41:35,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:41:35,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:35,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25616.96 MB 2025-02-15 04:41:35,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34055.99 MB 2025-02-15 04:41:35,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:41:35,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-15 04:41:35,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39296.43 MB 2025-02-15 04:41:35,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:41:35,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34055.99 MB 2025-02-15 04:41:35,826 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:41:35,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:35,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:41:35,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:35,828 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:41:35,833 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:41:35,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:35,834 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:41:35,834 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-15 04:41:44,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:44,829 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:41:44,837 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:41:44,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:44,844 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 115, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:41:44,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:44,846 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 115, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:41:46,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:41:46,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:41:46,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.87 seconds 2025-02-15 04:41:46,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:46,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21934.38 MB 2025-02-15 04:41:46,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22341.36 MB 2025-02-15 04:41:46,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.98 MB 2025-02-15 04:41:46,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51881.44 MB 2025-02-15 04:41:46,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24050.14 MB 2025-02-15 04:41:46,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27831.30 MB 2025-02-15 04:41:46,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31179.26 MB 2025-02-15 04:41:46,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:41:46,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:41:46,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:41:46,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:46,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22341.36 MB 2025-02-15 04:41:46,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22538.54 MB 2025-02-15 04:41:46,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.18 MB 2025-02-15 04:41:46,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24050.14 MB 2025-02-15 04:41:46,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24050.14 MB 2025-02-15 04:41:46,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:46,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23149.07 MB 2025-02-15 04:41:47,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:41:47,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:41:47,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.57 seconds 2025-02-15 04:41:47,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22538.54 MB 2025-02-15 04:41:47,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22691.16 MB 2025-02-15 04:41:47,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 152.62 MB 2025-02-15 04:41:47,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24050.14 MB 2025-02-15 04:41:47,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23773.32 MB 2025-02-15 04:41:47,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -276.82 MB 2025-02-15 04:41:47,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26625.33 MB 2025-02-15 04:41:47,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:41:47,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:41:47,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:41:47,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22691.09 MB 2025-02-15 04:41:47,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23234.20 MB 2025-02-15 04:41:47,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 543.11 MB 2025-02-15 04:41:47,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23773.32 MB 2025-02-15 04:41:47,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24593.30 MB 2025-02-15 04:41:47,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 819.99 MB 2025-02-15 04:41:47,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23641.72 MB 2025-02-15 04:41:47,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:41:47,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:41:47,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:41:47,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23234.20 MB 2025-02-15 04:41:47,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23893.86 MB 2025-02-15 04:41:47,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 659.65 MB 2025-02-15 04:41:47,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24593.30 MB 2025-02-15 04:41:47,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26229.08 MB 2025-02-15 04:41:47,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1635.78 MB 2025-02-15 04:41:47,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25474.03 MB 2025-02-15 04:41:47,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:41:47,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:41:47,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 04:41:47,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22691.09 MB 2025-02-15 04:41:47,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23893.86 MB 2025-02-15 04:41:47,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1202.76 MB 2025-02-15 04:41:47,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23773.32 MB 2025-02-15 04:41:47,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26229.08 MB 2025-02-15 04:41:47,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2455.76 MB 2025-02-15 04:41:47,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25474.03 MB 2025-02-15 04:41:47,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:41:47,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:41:47,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:41:47,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24531.14 MB 2025-02-15 04:41:47,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24808.83 MB 2025-02-15 04:41:47,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.69 MB 2025-02-15 04:41:47,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26229.08 MB 2025-02-15 04:41:47,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26409.44 MB 2025-02-15 04:41:47,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 04:41:47,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25012.32 MB 2025-02-15 04:41:47,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:41:47,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:41:47,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:41:47,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24984.07 MB 2025-02-15 04:41:47,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25213.68 MB 2025-02-15 04:41:47,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.61 MB 2025-02-15 04:41:47,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26409.44 MB 2025-02-15 04:41:47,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26409.44 MB 2025-02-15 04:41:47,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:41:47,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25213.68 MB 2025-02-15 04:41:47,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:41:47,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:41:47,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.64 seconds 2025-02-15 04:41:47,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21533.71 MB 2025-02-15 04:41:47,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25413.79 MB 2025-02-15 04:41:47,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3880.08 MB 2025-02-15 04:41:47,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51881.44 MB 2025-02-15 04:41:47,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26409.44 MB 2025-02-15 04:41:47,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25472.01 MB 2025-02-15 04:41:47,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25413.79 MB 2025-02-15 04:41:47,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:41:47,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:41:47,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 04:41:47,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25413.79 MB 2025-02-15 04:41:47,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28413.45 MB 2025-02-15 04:41:47,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2999.66 MB 2025-02-15 04:41:47,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26409.44 MB 2025-02-15 04:41:47,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30167.53 MB 2025-02-15 04:41:47,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3758.10 MB 2025-02-15 04:41:47,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28715.18 MB 2025-02-15 04:41:47,792 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 04:41:47,792 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:41:47,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:41:47,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:41:47,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:41:47,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:41:47,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28413.45 MB 2025-02-15 04:41:47,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36812.84 MB 2025-02-15 04:41:47,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-15 04:41:47,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30167.53 MB 2025-02-15 04:41:47,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40607.15 MB 2025-02-15 04:41:47,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-15 04:41:47,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36812.84 MB 2025-02-15 04:41:47,962 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 04:41:47,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:47,964 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:41:47,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:47,965 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:41:47,970 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:41:47,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:41:47,971 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:41:47,971 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:42:48,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:48,720 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:42:48,729 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:42:48,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:48,737 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 144, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:42:48,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:48,739 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 144, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:42:51,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:42:51,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:42:51,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.27 seconds 2025-02-15 04:42:51,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30473.25 MB 2025-02-15 04:42:51,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30982.86 MB 2025-02-15 04:42:51,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 509.61 MB 2025-02-15 04:42:51,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48960.11 MB 2025-02-15 04:42:51,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-15 04:42:51,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16152.26 MB 2025-02-15 04:42:51,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39944.62 MB 2025-02-15 04:42:51,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:42:51,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:42:51,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:42:51,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30982.86 MB 2025-02-15 04:42:51,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31230.32 MB 2025-02-15 04:42:51,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 247.46 MB 2025-02-15 04:42:51,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32807.85 MB 2025-02-15 04:42:51,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34321.99 MB 2025-02-15 04:42:51,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1514.14 MB 2025-02-15 04:42:51,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33006.67 MB 2025-02-15 04:42:51,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:42:51,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:42:51,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 04:42:51,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31230.32 MB 2025-02-15 04:42:51,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31421.42 MB 2025-02-15 04:42:51,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.10 MB 2025-02-15 04:42:51,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34321.99 MB 2025-02-15 04:42:51,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -757.07 MB 2025-02-15 04:42:51,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35401.01 MB 2025-02-15 04:42:51,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:42:51,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:42:51,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:42:51,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23084.57 MB 2025-02-15 04:42:51,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23764.64 MB 2025-02-15 04:42:51,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 680.07 MB 2025-02-15 04:42:51,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:51,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:51,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24274.92 MB 2025-02-15 04:42:51,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:42:51,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:42:51,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:42:51,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23764.64 MB 2025-02-15 04:42:51,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24571.74 MB 2025-02-15 04:42:51,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.11 MB 2025-02-15 04:42:51,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:51,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:51,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26567.65 MB 2025-02-15 04:42:51,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:42:51,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:42:51,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:42:51,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23084.57 MB 2025-02-15 04:42:51,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24571.74 MB 2025-02-15 04:42:51,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1487.18 MB 2025-02-15 04:42:51,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:51,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:51,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26567.65 MB 2025-02-15 04:42:51,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:42:51,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:42:51,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:42:51,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25123.82 MB 2025-02-15 04:42:51,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25399.94 MB 2025-02-15 04:42:51,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.12 MB 2025-02-15 04:42:51,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:51,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:51,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25666.91 MB 2025-02-15 04:42:51,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:42:51,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:42:51,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:42:51,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25548.59 MB 2025-02-15 04:42:51,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25776.52 MB 2025-02-15 04:42:51,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.93 MB 2025-02-15 04:42:51,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:51,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:51,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25777.25 MB 2025-02-15 04:42:51,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:42:51,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:42:51,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.17 seconds 2025-02-15 04:42:51,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:51,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29971.54 MB 2025-02-15 04:42:51,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25977.47 MB 2025-02-15 04:42:51,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3994.07 MB 2025-02-15 04:42:51,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48960.11 MB 2025-02-15 04:42:51,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:51,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15395.19 MB 2025-02-15 04:42:51,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25977.47 MB 2025-02-15 04:42:52,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:42:52,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:42:52,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:42:52,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:52,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25977.47 MB 2025-02-15 04:42:52,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28989.66 MB 2025-02-15 04:42:52,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.19 MB 2025-02-15 04:42:52,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:52,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33564.92 MB 2025-02-15 04:42:52,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:42:52,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29290.84 MB 2025-02-15 04:42:52,193 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 04:42:52,194 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 04:42:52,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:42:52,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:42:52,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:42:52,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:42:52,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28989.66 MB 2025-02-15 04:42:52,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37424.28 MB 2025-02-15 04:42:52,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 04:42:52,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33564.92 MB 2025-02-15 04:42:52,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41949.33 MB 2025-02-15 04:42:52,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 04:42:52,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37424.28 MB 2025-02-15 04:42:52,361 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 04:42:52,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:52,363 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:42:52,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:52,364 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:42:52,368 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:42:52,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:42:52,369 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:42:52,370 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 04:43:54,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:43:54,537 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:43:54,543 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:43:54,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:43:54,547 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1421, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:43:54,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:43:54,548 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1421, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:44:16,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:44:16,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:44:16,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.79 seconds 2025-02-15 04:44:16,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:16,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39998.53 MB 2025-02-15 04:44:16,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45027.50 MB 2025-02-15 04:44:16,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5028.97 MB 2025-02-15 04:44:16,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50333.75 MB 2025-02-15 04:44:16,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53416.56 MB 2025-02-15 04:44:16,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3082.81 MB 2025-02-15 04:44:16,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54000.01 MB 2025-02-15 04:44:16,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:44:16,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:44:16,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:44:16,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:16,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45027.50 MB 2025-02-15 04:44:16,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40293.25 MB 2025-02-15 04:44:16,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4734.25 MB 2025-02-15 04:44:16,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53416.56 MB 2025-02-15 04:44:16,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63027.81 MB 2025-02-15 04:44:16,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9611.25 MB 2025-02-15 04:44:16,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59478.23 MB 2025-02-15 04:44:18,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:44:18,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:44:18,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:44:18,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40293.25 MB 2025-02-15 04:44:18,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40824.09 MB 2025-02-15 04:44:18,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:44:18,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63027.81 MB 2025-02-15 04:44:18,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44195.38 MB 2025-02-15 04:44:18,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18832.42 MB 2025-02-15 04:44:18,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44802.64 MB 2025-02-15 04:44:18,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:44:18,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:44:18,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:44:18,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40824.09 MB 2025-02-15 04:44:18,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42713.63 MB 2025-02-15 04:44:18,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:44:18,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44195.38 MB 2025-02-15 04:44:18,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46082.82 MB 2025-02-15 04:44:18,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 04:44:18,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44131.06 MB 2025-02-15 04:44:18,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:44:18,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:44:18,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:44:18,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42713.63 MB 2025-02-15 04:44:18,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44956.53 MB 2025-02-15 04:44:18,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 04:44:18,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46082.82 MB 2025-02-15 04:44:18,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52810.48 MB 2025-02-15 04:44:18,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6727.66 MB 2025-02-15 04:44:18,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50500.81 MB 2025-02-15 04:44:18,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:44:18,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:44:18,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:44:18,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40824.09 MB 2025-02-15 04:44:18,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44956.53 MB 2025-02-15 04:44:18,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 04:44:18,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44195.38 MB 2025-02-15 04:44:18,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52810.48 MB 2025-02-15 04:44:18,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8615.10 MB 2025-02-15 04:44:18,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50500.81 MB 2025-02-15 04:44:18,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:44:18,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:44:18,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:44:18,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46490.07 MB 2025-02-15 04:44:18,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47257.08 MB 2025-02-15 04:44:18,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:44:18,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52810.48 MB 2025-02-15 04:44:18,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53229.91 MB 2025-02-15 04:44:18,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:44:18,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47964.86 MB 2025-02-15 04:44:18,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:44:18,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:44:18,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:44:18,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47669.96 MB 2025-02-15 04:44:18,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47898.93 MB 2025-02-15 04:44:18,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 04:44:18,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53229.91 MB 2025-02-15 04:44:18,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53229.91 MB 2025-02-15 04:44:18,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:44:18,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48112.86 MB 2025-02-15 04:44:18,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:44:18,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:44:18,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.23 seconds 2025-02-15 04:44:18,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:18,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35047.65 MB 2025-02-15 04:44:18,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48099.80 MB 2025-02-15 04:44:18,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13052.15 MB 2025-02-15 04:44:18,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50333.75 MB 2025-02-15 04:44:18,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53229.91 MB 2025-02-15 04:44:18,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2896.17 MB 2025-02-15 04:44:18,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48112.86 MB 2025-02-15 04:44:19,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:44:19,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:44:19,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:44:19,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:19,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48099.80 MB 2025-02-15 04:44:19,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40048.99 MB 2025-02-15 04:44:19,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8050.81 MB 2025-02-15 04:44:19,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53229.91 MB 2025-02-15 04:44:19,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53229.91 MB 2025-02-15 04:44:19,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:44:19,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50609.01 MB 2025-02-15 04:44:19,071 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 04:44:19,071 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:44:19,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:44:19,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:44:19,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:44:19,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:44:19,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40048.99 MB 2025-02-15 04:44:19,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48479.42 MB 2025-02-15 04:44:19,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.43 MB 2025-02-15 04:44:19,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53229.91 MB 2025-02-15 04:44:19,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57422.12 MB 2025-02-15 04:44:19,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 04:44:19,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48479.42 MB 2025-02-15 04:44:19,240 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 04:44:19,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:19,241 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:44:19,242 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:19,242 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:44:19,247 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:44:19,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:19,248 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:44:19,248 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:44:53,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:53,881 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:44:53,886 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:44:53,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:53,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1627, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:44:53,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:44:53,890 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1627, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:45:19,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:45:19,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:45:19,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.22 seconds 2025-02-15 04:45:19,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:19,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41433.97 MB 2025-02-15 04:45:19,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47192.75 MB 2025-02-15 04:45:19,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5758.78 MB 2025-02-15 04:45:19,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65802.34 MB 2025-02-15 04:45:19,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54142.17 MB 2025-02-15 04:45:19,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11660.17 MB 2025-02-15 04:45:19,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56115.47 MB 2025-02-15 04:45:19,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:45:19,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:45:19,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:45:19,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:19,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47192.75 MB 2025-02-15 04:45:19,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41364.18 MB 2025-02-15 04:45:19,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5828.57 MB 2025-02-15 04:45:19,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54142.17 MB 2025-02-15 04:45:19,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62176.36 MB 2025-02-15 04:45:19,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8034.19 MB 2025-02-15 04:45:19,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58034.40 MB 2025-02-15 04:45:21,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:45:21,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:45:21,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:45:21,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41364.18 MB 2025-02-15 04:45:21,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41895.02 MB 2025-02-15 04:45:21,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:45:21,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62176.36 MB 2025-02-15 04:45:21,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49798.97 MB 2025-02-15 04:45:21,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12377.39 MB 2025-02-15 04:45:21,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45873.57 MB 2025-02-15 04:45:21,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:45:21,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:45:21,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:45:21,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41895.02 MB 2025-02-15 04:45:21,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43784.56 MB 2025-02-15 04:45:21,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:45:21,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49798.97 MB 2025-02-15 04:45:21,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49798.97 MB 2025-02-15 04:45:21,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:45:21,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45201.99 MB 2025-02-15 04:45:21,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:45:21,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:45:21,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:45:21,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43784.56 MB 2025-02-15 04:45:21,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46026.41 MB 2025-02-15 04:45:21,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:45:21,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49798.97 MB 2025-02-15 04:45:21,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54045.70 MB 2025-02-15 04:45:21,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 04:45:21,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51570.69 MB 2025-02-15 04:45:21,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:45:21,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:45:21,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:45:21,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41895.02 MB 2025-02-15 04:45:21,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46026.41 MB 2025-02-15 04:45:21,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:45:21,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49798.97 MB 2025-02-15 04:45:21,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54045.70 MB 2025-02-15 04:45:21,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 04:45:21,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51570.69 MB 2025-02-15 04:45:21,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:45:21,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:45:21,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:45:21,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47559.96 MB 2025-02-15 04:45:21,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48326.96 MB 2025-02-15 04:45:21,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:45:21,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54045.70 MB 2025-02-15 04:45:21,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54465.13 MB 2025-02-15 04:45:21,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:45:21,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49034.75 MB 2025-02-15 04:45:21,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:45:21,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:45:21,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:45:21,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48739.85 MB 2025-02-15 04:45:21,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48971.81 MB 2025-02-15 04:45:21,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.96 MB 2025-02-15 04:45:21,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54465.13 MB 2025-02-15 04:45:21,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54465.13 MB 2025-02-15 04:45:21,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:45:21,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49184.83 MB 2025-02-15 04:45:21,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:45:21,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:45:21,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.67 seconds 2025-02-15 04:45:21,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35765.37 MB 2025-02-15 04:45:21,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49172.88 MB 2025-02-15 04:45:21,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13407.51 MB 2025-02-15 04:45:21,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65802.34 MB 2025-02-15 04:45:21,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54465.13 MB 2025-02-15 04:45:21,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11337.20 MB 2025-02-15 04:45:21,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49184.83 MB 2025-02-15 04:45:21,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:45:21,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:45:21,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:45:21,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49172.88 MB 2025-02-15 04:45:21,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40769.76 MB 2025-02-15 04:45:21,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8403.12 MB 2025-02-15 04:45:21,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54465.13 MB 2025-02-15 04:45:21,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54465.13 MB 2025-02-15 04:45:21,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:45:21,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51684.55 MB 2025-02-15 04:45:21,864 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:45:21,864 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:45:21,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:45:21,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:45:21,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 04:45:21,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:45:21,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40769.76 MB 2025-02-15 04:45:21,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49208.78 MB 2025-02-15 04:45:21,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:45:21,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54465.13 MB 2025-02-15 04:45:21,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62855.84 MB 2025-02-15 04:45:21,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:45:21,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49208.78 MB 2025-02-15 04:45:22,036 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:45:22,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:45:22,038 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:45:22,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:45:22,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:45:22,043 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:45:22,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:45:22,044 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:45:22,044 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:46:30,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:30,127 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:46:30,132 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:46:30,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:30,136 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 944, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:46:30,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:30,137 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 944, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:46:44,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:46:44,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:46:44,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.62 seconds 2025-02-15 04:46:44,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:44,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36674.72 MB 2025-02-15 04:46:44,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40015.48 MB 2025-02-15 04:46:44,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3340.76 MB 2025-02-15 04:46:44,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75440.85 MB 2025-02-15 04:46:44,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47534.05 MB 2025-02-15 04:46:44,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27906.80 MB 2025-02-15 04:46:44,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48863.99 MB 2025-02-15 04:46:44,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:46:44,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:46:44,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:46:44,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:44,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40015.48 MB 2025-02-15 04:46:44,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37813.48 MB 2025-02-15 04:46:44,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2202.00 MB 2025-02-15 04:46:44,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47534.05 MB 2025-02-15 04:46:44,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52791.61 MB 2025-02-15 04:46:44,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5257.56 MB 2025-02-15 04:46:44,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49375.73 MB 2025-02-15 04:46:46,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:46:46,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:46:46,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:46:46,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:46,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37813.48 MB 2025-02-15 04:46:46,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38344.32 MB 2025-02-15 04:46:46,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:46:46,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52791.61 MB 2025-02-15 04:46:46,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45608.86 MB 2025-02-15 04:46:46,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7182.75 MB 2025-02-15 04:46:46,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42322.87 MB 2025-02-15 04:46:46,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:46:46,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:46:46,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:46:46,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:46,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38344.32 MB 2025-02-15 04:46:46,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40233.85 MB 2025-02-15 04:46:46,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:46:46,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45608.86 MB 2025-02-15 04:46:46,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45608.86 MB 2025-02-15 04:46:46,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:46:46,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41651.28 MB 2025-02-15 04:46:46,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:46:46,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:46:46,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:46:46,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:46,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40233.85 MB 2025-02-15 04:46:46,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42475.71 MB 2025-02-15 04:46:46,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:46:46,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45608.86 MB 2025-02-15 04:46:46,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50327.45 MB 2025-02-15 04:46:46,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 04:46:46,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48019.99 MB 2025-02-15 04:46:46,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:46:46,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:46:46,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:46:46,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:46,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38344.32 MB 2025-02-15 04:46:46,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42475.71 MB 2025-02-15 04:46:46,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:46:46,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45608.86 MB 2025-02-15 04:46:46,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50327.45 MB 2025-02-15 04:46:46,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 04:46:46,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48019.99 MB 2025-02-15 04:46:47,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:46:47,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:46:47,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 04:46:47,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:47,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44009.25 MB 2025-02-15 04:46:47,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44776.25 MB 2025-02-15 04:46:47,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:46:47,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50327.45 MB 2025-02-15 04:46:47,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50746.88 MB 2025-02-15 04:46:47,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:46:47,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45484.04 MB 2025-02-15 04:46:47,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:46:47,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:46:47,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:46:47,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:47,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45189.14 MB 2025-02-15 04:46:47,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45418.42 MB 2025-02-15 04:46:47,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.28 MB 2025-02-15 04:46:47,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50746.88 MB 2025-02-15 04:46:47,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50746.88 MB 2025-02-15 04:46:47,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:46:47,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45625.93 MB 2025-02-15 04:46:47,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:46:47,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:46:47,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.03 seconds 2025-02-15 04:46:47,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:47,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33385.74 MB 2025-02-15 04:46:47,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45619.50 MB 2025-02-15 04:46:47,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12233.75 MB 2025-02-15 04:46:47,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75440.85 MB 2025-02-15 04:46:47,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50746.88 MB 2025-02-15 04:46:47,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24693.96 MB 2025-02-15 04:46:47,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45625.93 MB 2025-02-15 04:46:47,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:46:47,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:46:47,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:46:47,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:47,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45619.50 MB 2025-02-15 04:46:47,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38390.13 MB 2025-02-15 04:46:47,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7229.36 MB 2025-02-15 04:46:47,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50746.88 MB 2025-02-15 04:46:47,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50746.88 MB 2025-02-15 04:46:47,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:46:47,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48131.16 MB 2025-02-15 04:46:47,452 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:46:47,452 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:46:47,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:46:47,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:46:47,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:46:47,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:46:47,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38390.13 MB 2025-02-15 04:46:47,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46829.16 MB 2025-02-15 04:46:47,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:46:47,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50746.88 MB 2025-02-15 04:46:47,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59137.59 MB 2025-02-15 04:46:47,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:46:47,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46829.16 MB 2025-02-15 04:46:47,621 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:46:47,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:47,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:46:47,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:47,623 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:46:47,628 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:46:47,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:46:47,629 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:46:47,629 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:47:00,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:00,861 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:47:00,866 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:47:00,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:00,870 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:47:00,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:00,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:47:26,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:47:26,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:47:26,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.06 seconds 2025-02-15 04:47:26,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:26,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41782.38 MB 2025-02-15 04:47:26,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47717.32 MB 2025-02-15 04:47:26,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-15 04:47:26,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71722.60 MB 2025-02-15 04:47:26,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54322.53 MB 2025-02-15 04:47:26,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17400.07 MB 2025-02-15 04:47:26,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56689.56 MB 2025-02-15 04:47:27,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:47:27,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:47:27,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:47:27,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:27,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47717.32 MB 2025-02-15 04:47:27,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41624.12 MB 2025-02-15 04:47:27,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-15 04:47:27,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54322.53 MB 2025-02-15 04:47:27,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66739.77 MB 2025-02-15 04:47:27,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12417.24 MB 2025-02-15 04:47:27,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64261.21 MB 2025-02-15 04:47:28,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:47:28,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:47:28,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:47:28,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:28,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41624.12 MB 2025-02-15 04:47:28,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42154.96 MB 2025-02-15 04:47:28,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:47:28,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66739.77 MB 2025-02-15 04:47:28,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49803.17 MB 2025-02-15 04:47:28,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16936.60 MB 2025-02-15 04:47:28,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46133.50 MB 2025-02-15 04:47:29,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:47:29,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:47:29,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:47:29,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42154.96 MB 2025-02-15 04:47:29,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44044.49 MB 2025-02-15 04:47:29,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:47:29,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49803.17 MB 2025-02-15 04:47:29,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49803.17 MB 2025-02-15 04:47:29,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:47:29,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45461.92 MB 2025-02-15 04:47:29,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:47:29,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:47:29,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:47:29,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44044.49 MB 2025-02-15 04:47:29,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46286.35 MB 2025-02-15 04:47:29,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:47:29,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49803.17 MB 2025-02-15 04:47:29,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54521.76 MB 2025-02-15 04:47:29,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 04:47:29,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51830.63 MB 2025-02-15 04:47:29,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:47:29,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:47:29,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:47:29,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42154.96 MB 2025-02-15 04:47:29,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46286.35 MB 2025-02-15 04:47:29,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:47:29,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49803.17 MB 2025-02-15 04:47:29,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54521.76 MB 2025-02-15 04:47:29,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 04:47:29,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51830.63 MB 2025-02-15 04:47:29,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:47:29,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:47:29,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:47:29,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47819.89 MB 2025-02-15 04:47:29,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48586.89 MB 2025-02-15 04:47:29,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:47:29,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54521.76 MB 2025-02-15 04:47:29,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54941.19 MB 2025-02-15 04:47:29,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:47:29,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49294.68 MB 2025-02-15 04:47:29,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:47:29,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:47:29,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:47:29,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48999.78 MB 2025-02-15 04:47:29,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49228.52 MB 2025-02-15 04:47:29,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-15 04:47:29,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54941.19 MB 2025-02-15 04:47:29,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54941.19 MB 2025-02-15 04:47:29,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:47:29,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49436.66 MB 2025-02-15 04:47:29,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:47:29,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:47:29,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.53 seconds 2025-02-15 04:47:29,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35939.57 MB 2025-02-15 04:47:29,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49429.00 MB 2025-02-15 04:47:29,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13489.43 MB 2025-02-15 04:47:29,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71722.60 MB 2025-02-15 04:47:29,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54941.19 MB 2025-02-15 04:47:29,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16781.41 MB 2025-02-15 04:47:29,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49436.66 MB 2025-02-15 04:47:29,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:47:29,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:47:29,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:47:29,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49429.00 MB 2025-02-15 04:47:29,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40934.82 MB 2025-02-15 04:47:29,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8494.18 MB 2025-02-15 04:47:29,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54941.19 MB 2025-02-15 04:47:29,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54941.19 MB 2025-02-15 04:47:29,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:47:29,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51933.30 MB 2025-02-15 04:47:29,686 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 04:47:29,687 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:47:29,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:47:29,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:47:29,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:47:29,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:47:29,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40934.82 MB 2025-02-15 04:47:29,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49348.80 MB 2025-02-15 04:47:29,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 04:47:29,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54941.19 MB 2025-02-15 04:47:29,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63306.73 MB 2025-02-15 04:47:29,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 04:47:29,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49348.80 MB 2025-02-15 04:47:29,850 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 04:47:29,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:29,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:47:29,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:29,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:47:29,857 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:47:29,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:47:29,858 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:47:29,858 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:49:17,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:17,648 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:49:17,653 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:49:17,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:17,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 252, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:49:17,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:17,658 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 252, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:49:21,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:49:21,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:49:21,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.89 seconds 2025-02-15 04:49:21,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:21,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31852.75 MB 2025-02-15 04:49:21,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32744.56 MB 2025-02-15 04:49:21,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 891.81 MB 2025-02-15 04:49:21,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75853.99 MB 2025-02-15 04:49:21,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39233.52 MB 2025-02-15 04:49:21,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36620.47 MB 2025-02-15 04:49:21,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41550.61 MB 2025-02-15 04:49:21,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:49:21,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:49:21,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:49:21,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:21,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32744.56 MB 2025-02-15 04:49:21,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33176.58 MB 2025-02-15 04:49:21,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.02 MB 2025-02-15 04:49:21,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39233.52 MB 2025-02-15 04:49:21,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39233.52 MB 2025-02-15 04:49:21,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:49:21,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36284.16 MB 2025-02-15 04:49:22,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:49:22,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:49:22,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.19 seconds 2025-02-15 04:49:22,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:22,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33176.58 MB 2025-02-15 04:49:22,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33511.01 MB 2025-02-15 04:49:22,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 334.43 MB 2025-02-15 04:49:22,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39233.52 MB 2025-02-15 04:49:22,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38761.66 MB 2025-02-15 04:49:22,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 04:49:22,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37431.16 MB 2025-02-15 04:49:22,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:49:22,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:49:22,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:49:22,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:22,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33511.01 MB 2025-02-15 04:49:22,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34701.13 MB 2025-02-15 04:49:22,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1190.12 MB 2025-02-15 04:49:22,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38761.66 MB 2025-02-15 04:49:22,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38763.76 MB 2025-02-15 04:49:22,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 04:49:22,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35594.11 MB 2025-02-15 04:49:22,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:49:22,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:49:22,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 04:49:22,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:22,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34701.13 MB 2025-02-15 04:49:22,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36113.52 MB 2025-02-15 04:49:22,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1412.39 MB 2025-02-15 04:49:22,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38763.76 MB 2025-02-15 04:49:22,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41443.92 MB 2025-02-15 04:49:22,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2680.16 MB 2025-02-15 04:49:22,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39611.11 MB 2025-02-15 04:49:22,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:49:22,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:49:22,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 04:49:22,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:22,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33511.01 MB 2025-02-15 04:49:22,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36113.52 MB 2025-02-15 04:49:22,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2602.51 MB 2025-02-15 04:49:22,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38761.66 MB 2025-02-15 04:49:22,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41443.92 MB 2025-02-15 04:49:22,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2682.26 MB 2025-02-15 04:49:22,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39611.11 MB 2025-02-15 04:49:23,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:49:23,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:49:23,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:49:23,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:23,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37079.65 MB 2025-02-15 04:49:23,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37562.86 MB 2025-02-15 04:49:23,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 483.21 MB 2025-02-15 04:49:23,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41443.92 MB 2025-02-15 04:49:23,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41706.06 MB 2025-02-15 04:49:23,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 262.14 MB 2025-02-15 04:49:23,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38008.77 MB 2025-02-15 04:49:23,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:49:23,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:49:23,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:49:23,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:23,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37822.99 MB 2025-02-15 04:49:23,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38037.19 MB 2025-02-15 04:49:23,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.21 MB 2025-02-15 04:49:23,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41706.06 MB 2025-02-15 04:49:23,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41706.06 MB 2025-02-15 04:49:23,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:49:23,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38141.17 MB 2025-02-15 04:49:23,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:49:23,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:49:23,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.38 seconds 2025-02-15 04:49:23,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:23,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30974.76 MB 2025-02-15 04:49:23,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38238.26 MB 2025-02-15 04:49:23,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7263.50 MB 2025-02-15 04:49:23,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75853.99 MB 2025-02-15 04:49:23,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41706.06 MB 2025-02-15 04:49:23,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34147.93 MB 2025-02-15 04:49:23,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38238.26 MB 2025-02-15 04:49:23,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:49:23,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:49:23,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 04:49:23,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:23,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38238.26 MB 2025-02-15 04:49:23,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41252.30 MB 2025-02-15 04:49:23,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:49:23,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41706.06 MB 2025-02-15 04:49:23,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43316.67 MB 2025-02-15 04:49:23,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1610.61 MB 2025-02-15 04:49:23,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41553.93 MB 2025-02-15 04:49:23,339 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:49:23,340 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:49:23,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:49:23,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:49:23,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:49:23,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:49:23,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35280.65 MB 2025-02-15 04:49:23,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43719.67 MB 2025-02-15 04:49:23,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:49:23,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43316.67 MB 2025-02-15 04:49:23,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53806.63 MB 2025-02-15 04:49:23,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:49:23,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43719.67 MB 2025-02-15 04:49:23,596 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:49:23,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:23,599 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:49:23,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:23,601 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:49:23,609 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:49:23,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:23,611 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:49:23,611 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:49:32,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:32,663 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:49:32,668 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:49:32,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:32,671 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2373, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:49:32,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:49:32,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2373, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:50:09,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:50:09,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:50:09,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.78 seconds 2025-02-15 04:50:09,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:09,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46632.22 MB 2025-02-15 04:50:09,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55030.13 MB 2025-02-15 04:50:09,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.91 MB 2025-02-15 04:50:09,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74792.83 MB 2025-02-15 04:50:09,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60999.86 MB 2025-02-15 04:50:09,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13792.97 MB 2025-02-15 04:50:09,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64031.08 MB 2025-02-15 04:50:09,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:50:09,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:50:09,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:50:09,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:09,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 55030.13 MB 2025-02-15 04:50:09,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45242.40 MB 2025-02-15 04:50:09,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9787.73 MB 2025-02-15 04:50:09,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60999.86 MB 2025-02-15 04:50:09,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79012.30 MB 2025-02-15 04:50:09,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18012.44 MB 2025-02-15 04:50:09,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 79991.08 MB 2025-02-15 04:50:11,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:50:11,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:50:11,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:50:11,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:11,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45242.40 MB 2025-02-15 04:50:11,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45773.24 MB 2025-02-15 04:50:11,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:50:11,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79012.30 MB 2025-02-15 04:50:11,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49815.75 MB 2025-02-15 04:50:11,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29196.55 MB 2025-02-15 04:50:11,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49752.83 MB 2025-02-15 04:50:11,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:50:11,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:50:11,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:50:11,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:11,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45773.24 MB 2025-02-15 04:50:11,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47662.32 MB 2025-02-15 04:50:11,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.08 MB 2025-02-15 04:50:11,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49815.75 MB 2025-02-15 04:50:11,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50759.47 MB 2025-02-15 04:50:11,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:50:11,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49079.75 MB 2025-02-15 04:50:11,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:50:11,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:50:11,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:50:11,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:11,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47662.32 MB 2025-02-15 04:50:11,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49904.18 MB 2025-02-15 04:50:11,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:50:11,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50759.47 MB 2025-02-15 04:50:11,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57365.50 MB 2025-02-15 04:50:11,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:50:11,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55448.46 MB 2025-02-15 04:50:11,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:50:11,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:50:11,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:50:11,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:11,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45773.24 MB 2025-02-15 04:50:11,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49904.18 MB 2025-02-15 04:50:11,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.93 MB 2025-02-15 04:50:11,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49815.75 MB 2025-02-15 04:50:11,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57365.50 MB 2025-02-15 04:50:11,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 04:50:11,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55448.46 MB 2025-02-15 04:50:12,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:50:12,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:50:12,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:50:12,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:12,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51437.72 MB 2025-02-15 04:50:12,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52204.72 MB 2025-02-15 04:50:12,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:50:12,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57365.50 MB 2025-02-15 04:50:12,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57784.93 MB 2025-02-15 04:50:12,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:50:12,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52912.51 MB 2025-02-15 04:50:12,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:50:12,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:50:12,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:50:12,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:12,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52617.61 MB 2025-02-15 04:50:12,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52846.05 MB 2025-02-15 04:50:12,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 04:50:12,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57784.93 MB 2025-02-15 04:50:12,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57784.93 MB 2025-02-15 04:50:12,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:12,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53090.34 MB 2025-02-15 04:50:12,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:50:12,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:50:12,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.36 seconds 2025-02-15 04:50:12,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:12,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38364.49 MB 2025-02-15 04:50:12,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53046.41 MB 2025-02-15 04:50:12,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14681.92 MB 2025-02-15 04:50:12,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70592.23 MB 2025-02-15 04:50:12,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57784.93 MB 2025-02-15 04:50:12,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12807.31 MB 2025-02-15 04:50:12,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53090.34 MB 2025-02-15 04:50:12,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:50:12,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:50:12,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:50:12,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:12,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53046.41 MB 2025-02-15 04:50:12,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43357.84 MB 2025-02-15 04:50:12,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9688.58 MB 2025-02-15 04:50:12,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57784.93 MB 2025-02-15 04:50:12,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57784.93 MB 2025-02-15 04:50:12,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:12,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55549.17 MB 2025-02-15 04:50:12,316 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 04:50:12,317 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-15 04:50:12,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:50:12,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:50:12,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:50:12,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:12,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43357.84 MB 2025-02-15 04:50:12,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51767.14 MB 2025-02-15 04:50:12,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 04:50:12,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57784.93 MB 2025-02-15 04:50:12,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66144.17 MB 2025-02-15 04:50:12,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 04:50:12,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51767.14 MB 2025-02-15 04:50:12,481 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 04:50:12,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:12,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:50:12,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:12,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:50:12,488 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:50:12,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:12,489 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:50:12,489 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-15 04:50:22,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:22,618 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:50:22,623 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:50:22,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:22,627 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:50:22,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:22,628 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:50:25,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:50:25,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:50:25,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.11 seconds 2025-02-15 04:50:25,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:25,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31469.50 MB 2025-02-15 04:50:25,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32166.67 MB 2025-02-15 04:50:25,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 697.17 MB 2025-02-15 04:50:25,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74503.42 MB 2025-02-15 04:50:25,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:50:25,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36805.02 MB 2025-02-15 04:50:25,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41167.36 MB 2025-02-15 04:50:25,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:50:25,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:50:25,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:50:25,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:25,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32166.67 MB 2025-02-15 04:50:25,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32377.97 MB 2025-02-15 04:50:25,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.30 MB 2025-02-15 04:50:25,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:50:25,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:50:25,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:25,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34680.91 MB 2025-02-15 04:50:26,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:50:26,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:50:26,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-15 04:50:26,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32377.97 MB 2025-02-15 04:50:26,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32615.52 MB 2025-02-15 04:50:26,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.55 MB 2025-02-15 04:50:26,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:50:26,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 04:50:26,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 04:50:26,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36548.66 MB 2025-02-15 04:50:26,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:50:26,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:50:26,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:50:26,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32615.52 MB 2025-02-15 04:50:26,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33460.88 MB 2025-02-15 04:50:26,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.36 MB 2025-02-15 04:50:26,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 04:50:26,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 04:50:26,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:26,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34095.19 MB 2025-02-15 04:50:26,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:50:26,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:50:26,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:50:26,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33460.88 MB 2025-02-15 04:50:26,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34464.15 MB 2025-02-15 04:50:26,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.27 MB 2025-02-15 04:50:26,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 04:50:26,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38661.00 MB 2025-02-15 04:50:26,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1906.31 MB 2025-02-15 04:50:26,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36945.84 MB 2025-02-15 04:50:26,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:50:26,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:50:26,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:50:26,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32615.52 MB 2025-02-15 04:50:26,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34464.15 MB 2025-02-15 04:50:26,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1848.63 MB 2025-02-15 04:50:26,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 04:50:26,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38661.00 MB 2025-02-15 04:50:26,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1906.31 MB 2025-02-15 04:50:26,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36945.84 MB 2025-02-15 04:50:26,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:50:26,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:50:26,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:50:26,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35150.41 MB 2025-02-15 04:50:26,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35494.30 MB 2025-02-15 04:50:26,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 343.89 MB 2025-02-15 04:50:26,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38661.00 MB 2025-02-15 04:50:26,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38847.64 MB 2025-02-15 04:50:26,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-15 04:50:26,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35818.15 MB 2025-02-15 04:50:26,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:50:26,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:50:26,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:50:26,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35679.07 MB 2025-02-15 04:50:26,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35888.25 MB 2025-02-15 04:50:26,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.17 MB 2025-02-15 04:50:26,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38847.64 MB 2025-02-15 04:50:26,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38847.64 MB 2025-02-15 04:50:26,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:26,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35921.92 MB 2025-02-15 04:50:26,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:50:26,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:50:26,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.19 seconds 2025-02-15 04:50:26,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:26,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30783.14 MB 2025-02-15 04:50:26,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36089.27 MB 2025-02-15 04:50:26,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5306.14 MB 2025-02-15 04:50:26,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74503.42 MB 2025-02-15 04:50:26,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38847.64 MB 2025-02-15 04:50:26,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35655.78 MB 2025-02-15 04:50:26,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36089.27 MB 2025-02-15 04:50:27,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:50:27,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:50:27,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:50:27,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:27,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36089.27 MB 2025-02-15 04:50:27,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34744.45 MB 2025-02-15 04:50:27,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1344.82 MB 2025-02-15 04:50:27,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38847.64 MB 2025-02-15 04:50:27,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38847.64 MB 2025-02-15 04:50:27,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:50:27,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36189.71 MB 2025-02-15 04:50:27,107 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 04:50:27,108 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 04:50:27,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:50:27,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:50:27,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:50:27,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:50:27,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34744.45 MB 2025-02-15 04:50:27,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43181.93 MB 2025-02-15 04:50:27,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 04:50:27,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38847.64 MB 2025-02-15 04:50:27,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47236.25 MB 2025-02-15 04:50:27,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 04:50:27,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43181.93 MB 2025-02-15 04:50:27,273 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 04:50:27,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:27,274 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:50:27,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:27,275 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:50:27,280 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:50:27,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:50:27,281 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:50:27,281 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 04:51:06,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:06,109 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:51:06,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:51:06,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:06,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:51:06,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:06,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:51:09,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:51:09,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:51:09,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.90 seconds 2025-02-15 04:51:09,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31378.91 MB 2025-02-15 04:51:09,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32030.08 MB 2025-02-15 04:51:09,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-15 04:51:09,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55624.86 MB 2025-02-15 04:51:09,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37081.84 MB 2025-02-15 04:51:09,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18543.02 MB 2025-02-15 04:51:09,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40850.28 MB 2025-02-15 04:51:09,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:51:09,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:51:09,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:51:09,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32030.08 MB 2025-02-15 04:51:09,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32219.15 MB 2025-02-15 04:51:09,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 189.07 MB 2025-02-15 04:51:09,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37081.84 MB 2025-02-15 04:51:09,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37081.84 MB 2025-02-15 04:51:09,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:51:09,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34361.79 MB 2025-02-15 04:51:09,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:51:09,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:51:09,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-15 04:51:09,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32219.15 MB 2025-02-15 04:51:09,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32439.45 MB 2025-02-15 04:51:09,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.30 MB 2025-02-15 04:51:09,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37081.84 MB 2025-02-15 04:51:09,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37060.87 MB 2025-02-15 04:51:09,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20.97 MB 2025-02-15 04:51:09,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36389.84 MB 2025-02-15 04:51:09,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:51:09,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:51:09,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:51:09,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32439.39 MB 2025-02-15 04:51:09,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33223.35 MB 2025-02-15 04:51:09,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 783.97 MB 2025-02-15 04:51:09,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37060.87 MB 2025-02-15 04:51:09,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37060.87 MB 2025-02-15 04:51:09,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:51:09,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33811.59 MB 2025-02-15 04:51:09,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:51:09,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:51:09,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:51:09,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33223.35 MB 2025-02-15 04:51:09,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34153.76 MB 2025-02-15 04:51:09,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 930.41 MB 2025-02-15 04:51:09,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37060.87 MB 2025-02-15 04:51:09,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-15 04:51:09,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 983.56 MB 2025-02-15 04:51:09,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36455.12 MB 2025-02-15 04:51:09,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:51:09,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:51:09,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:51:09,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:09,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32439.39 MB 2025-02-15 04:51:09,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34153.76 MB 2025-02-15 04:51:09,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.37 MB 2025-02-15 04:51:09,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37060.87 MB 2025-02-15 04:51:09,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-15 04:51:09,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 983.56 MB 2025-02-15 04:51:09,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36455.12 MB 2025-02-15 04:51:10,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:51:10,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:51:10,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:51:10,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:10,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34790.18 MB 2025-02-15 04:51:10,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35108.49 MB 2025-02-15 04:51:10,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.31 MB 2025-02-15 04:51:10,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38044.43 MB 2025-02-15 04:51:10,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38218.50 MB 2025-02-15 04:51:10,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 04:51:10,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35411.14 MB 2025-02-15 04:51:10,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:51:10,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:51:10,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:51:10,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:10,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35279.84 MB 2025-02-15 04:51:10,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35499.35 MB 2025-02-15 04:51:10,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.50 MB 2025-02-15 04:51:10,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38218.50 MB 2025-02-15 04:51:10,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38218.50 MB 2025-02-15 04:51:10,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:51:10,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35518.97 MB 2025-02-15 04:51:10,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:51:10,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:51:10,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.90 seconds 2025-02-15 04:51:10,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:10,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30737.84 MB 2025-02-15 04:51:10,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35700.27 MB 2025-02-15 04:51:10,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4962.43 MB 2025-02-15 04:51:10,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55624.86 MB 2025-02-15 04:51:10,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38218.50 MB 2025-02-15 04:51:10,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17406.36 MB 2025-02-15 04:51:10,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35700.27 MB 2025-02-15 04:51:10,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:51:10,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:51:10,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:51:10,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:10,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35700.27 MB 2025-02-15 04:51:10,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34635.68 MB 2025-02-15 04:51:10,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1064.59 MB 2025-02-15 04:51:10,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38218.50 MB 2025-02-15 04:51:10,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38218.50 MB 2025-02-15 04:51:10,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:51:10,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36302.63 MB 2025-02-15 04:51:10,312 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 04:51:10,312 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 04:51:10,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:51:10,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:51:10,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:51:10,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:51:10,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34635.68 MB 2025-02-15 04:51:10,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43068.98 MB 2025-02-15 04:51:10,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 04:51:10,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38218.50 MB 2025-02-15 04:51:10,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46602.91 MB 2025-02-15 04:51:10,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 04:51:10,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43068.98 MB 2025-02-15 04:51:10,482 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 04:51:10,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:10,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:51:10,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:10,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:51:10,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:51:10,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:10,490 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:51:10,490 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 04:51:58,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:58,357 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:51:58,362 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:51:58,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:58,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:51:58,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:51:58,367 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:52:13,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:52:13,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:52:13,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.17 seconds 2025-02-15 04:52:13,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:13,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36967.38 MB 2025-02-15 04:52:13,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40457.04 MB 2025-02-15 04:52:13,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3489.66 MB 2025-02-15 04:52:13,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54987.33 MB 2025-02-15 04:52:13,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47685.04 MB 2025-02-15 04:52:13,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7302.28 MB 2025-02-15 04:52:13,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49383.15 MB 2025-02-15 04:52:13,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:52:13,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:52:13,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:52:13,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:13,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40457.04 MB 2025-02-15 04:52:13,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38031.82 MB 2025-02-15 04:52:13,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2425.22 MB 2025-02-15 04:52:13,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47685.04 MB 2025-02-15 04:52:13,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55889.10 MB 2025-02-15 04:52:13,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8204.06 MB 2025-02-15 04:52:13,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51466.50 MB 2025-02-15 04:52:15,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:52:15,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:52:15,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:52:15,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38031.82 MB 2025-02-15 04:52:15,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38562.66 MB 2025-02-15 04:52:15,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:52:15,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55889.10 MB 2025-02-15 04:52:15,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45610.96 MB 2025-02-15 04:52:15,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10278.14 MB 2025-02-15 04:52:15,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42541.21 MB 2025-02-15 04:52:15,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:52:15,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:52:15,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:52:15,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38562.66 MB 2025-02-15 04:52:15,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40452.20 MB 2025-02-15 04:52:15,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:52:15,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45610.96 MB 2025-02-15 04:52:15,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45610.96 MB 2025-02-15 04:52:15,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:15,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41869.63 MB 2025-02-15 04:52:15,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:52:15,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:52:15,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:52:15,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40452.20 MB 2025-02-15 04:52:15,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42694.05 MB 2025-02-15 04:52:15,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:52:15,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45610.96 MB 2025-02-15 04:52:15,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50801.41 MB 2025-02-15 04:52:15,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 04:52:15,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48238.34 MB 2025-02-15 04:52:15,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:52:15,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:52:15,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:52:15,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38562.66 MB 2025-02-15 04:52:15,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42694.05 MB 2025-02-15 04:52:15,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:52:15,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45610.96 MB 2025-02-15 04:52:15,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50801.41 MB 2025-02-15 04:52:15,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 04:52:15,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48238.34 MB 2025-02-15 04:52:15,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:52:15,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:52:15,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 04:52:15,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44227.60 MB 2025-02-15 04:52:15,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44994.60 MB 2025-02-15 04:52:15,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:52:15,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50801.41 MB 2025-02-15 04:52:15,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51220.84 MB 2025-02-15 04:52:15,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:52:15,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45702.39 MB 2025-02-15 04:52:15,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:52:15,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:52:15,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:52:15,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45407.49 MB 2025-02-15 04:52:15,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45636.60 MB 2025-02-15 04:52:15,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-15 04:52:15,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51220.84 MB 2025-02-15 04:52:15,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51220.84 MB 2025-02-15 04:52:15,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:15,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45873.53 MB 2025-02-15 04:52:15,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:52:15,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:52:15,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.58 seconds 2025-02-15 04:52:15,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:15,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33532.07 MB 2025-02-15 04:52:15,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45836.68 MB 2025-02-15 04:52:15,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12304.61 MB 2025-02-15 04:52:15,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54987.33 MB 2025-02-15 04:52:15,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51220.84 MB 2025-02-15 04:52:15,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3766.48 MB 2025-02-15 04:52:15,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45873.53 MB 2025-02-15 04:52:16,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:52:16,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:52:16,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:52:16,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:16,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45836.68 MB 2025-02-15 04:52:16,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38521.23 MB 2025-02-15 04:52:16,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7315.46 MB 2025-02-15 04:52:16,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51220.84 MB 2025-02-15 04:52:16,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51220.84 MB 2025-02-15 04:52:16,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:16,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48336.06 MB 2025-02-15 04:52:16,237 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 04:52:16,237 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:52:16,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:52:16,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:52:16,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:52:16,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:16,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38521.23 MB 2025-02-15 04:52:16,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46918.63 MB 2025-02-15 04:52:16,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 04:52:16,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51220.84 MB 2025-02-15 04:52:16,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59571.70 MB 2025-02-15 04:52:16,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 04:52:16,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46918.63 MB 2025-02-15 04:52:16,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 04:52:16,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:16,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:52:16,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:16,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:52:16,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:52:16,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:16,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:52:16,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:52:25,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:25,748 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:52:25,753 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:52:25,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:25,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:52:25,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:25,757 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:52:43,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:52:43,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:52:43,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.12 seconds 2025-02-15 04:52:43,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:43,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38193.77 MB 2025-02-15 04:52:43,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42306.29 MB 2025-02-15 04:52:43,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4112.52 MB 2025-02-15 04:52:43,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67922.56 MB 2025-02-15 04:52:43,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48274.34 MB 2025-02-15 04:52:43,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19648.22 MB 2025-02-15 04:52:43,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51289.83 MB 2025-02-15 04:52:43,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:52:43,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:52:43,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:52:43,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:43,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42306.29 MB 2025-02-15 04:52:43,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38946.79 MB 2025-02-15 04:52:43,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3359.50 MB 2025-02-15 04:52:43,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48274.34 MB 2025-02-15 04:52:43,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57854.13 MB 2025-02-15 04:52:43,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9579.79 MB 2025-02-15 04:52:43,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54692.09 MB 2025-02-15 04:52:45,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:52:45,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:52:45,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 04:52:45,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:45,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38946.79 MB 2025-02-15 04:52:45,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39477.63 MB 2025-02-15 04:52:45,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:52:45,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57854.13 MB 2025-02-15 04:52:45,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47571.80 MB 2025-02-15 04:52:45,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10282.34 MB 2025-02-15 04:52:45,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43456.18 MB 2025-02-15 04:52:45,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:52:45,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:52:45,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:52:45,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:45,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39477.63 MB 2025-02-15 04:52:45,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41367.17 MB 2025-02-15 04:52:45,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:52:45,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47571.80 MB 2025-02-15 04:52:45,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47571.80 MB 2025-02-15 04:52:45,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:45,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42784.60 MB 2025-02-15 04:52:46,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:52:46,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:52:46,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:52:46,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41367.17 MB 2025-02-15 04:52:46,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43609.02 MB 2025-02-15 04:52:46,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:52:46,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47571.80 MB 2025-02-15 04:52:46,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51346.67 MB 2025-02-15 04:52:46,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 04:52:46,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49153.31 MB 2025-02-15 04:52:46,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:52:46,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:52:46,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:52:46,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39477.63 MB 2025-02-15 04:52:46,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43609.02 MB 2025-02-15 04:52:46,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:52:46,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47571.80 MB 2025-02-15 04:52:46,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51346.67 MB 2025-02-15 04:52:46,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 04:52:46,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49153.31 MB 2025-02-15 04:52:46,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:52:46,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:52:46,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:52:46,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45142.57 MB 2025-02-15 04:52:46,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45909.57 MB 2025-02-15 04:52:46,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:52:46,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51346.67 MB 2025-02-15 04:52:46,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-15 04:52:46,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:52:46,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46617.36 MB 2025-02-15 04:52:46,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:52:46,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:52:46,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:52:46,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46322.46 MB 2025-02-15 04:52:46,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46551.37 MB 2025-02-15 04:52:46,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 04:52:46,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51766.10 MB 2025-02-15 04:52:46,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-15 04:52:46,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:46,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46778.24 MB 2025-02-15 04:52:46,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:52:46,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:52:46,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.55 seconds 2025-02-15 04:52:46,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34145.27 MB 2025-02-15 04:52:46,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46752.20 MB 2025-02-15 04:52:46,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12606.92 MB 2025-02-15 04:52:46,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67922.56 MB 2025-02-15 04:52:46,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-15 04:52:46,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16156.46 MB 2025-02-15 04:52:46,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46778.24 MB 2025-02-15 04:52:46,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:52:46,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:52:46,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:52:46,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46752.20 MB 2025-02-15 04:52:46,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39145.85 MB 2025-02-15 04:52:46,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7606.34 MB 2025-02-15 04:52:46,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51766.10 MB 2025-02-15 04:52:46,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-15 04:52:46,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:52:46,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49260.79 MB 2025-02-15 04:52:46,601 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 04:52:46,601 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:52:46,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:52:46,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:52:46,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:52:46,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:52:46,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39145.85 MB 2025-02-15 04:52:46,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47574.98 MB 2025-02-15 04:52:46,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 04:52:46,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51766.10 MB 2025-02-15 04:52:46,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60146.32 MB 2025-02-15 04:52:46,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 04:52:46,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47574.98 MB 2025-02-15 04:52:46,771 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 04:52:46,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:46,773 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:52:46,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:46,774 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:52:46,778 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:52:46,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:46,780 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:52:46,780 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 04:52:59,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:59,237 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:52:59,242 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:52:59,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:59,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:52:59,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:52:59,246 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:53:02,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:53:02,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:53:02,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-15 04:53:02,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31364.98 MB 2025-02-15 04:53:02,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32009.06 MB 2025-02-15 04:53:02,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-15 04:53:02,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68526.54 MB 2025-02-15 04:53:02,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27596.42 MB 2025-02-15 04:53:02,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40836.35 MB 2025-02-15 04:53:02,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:53:02,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:53:02,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:53:02,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32009.06 MB 2025-02-15 04:53:02,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31478.36 MB 2025-02-15 04:53:02,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -530.70 MB 2025-02-15 04:53:02,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32879.98 MB 2025-02-15 04:53:02,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:53:02,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:53:02,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.31 seconds 2025-02-15 04:53:02,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31478.36 MB 2025-02-15 04:53:02,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31560.64 MB 2025-02-15 04:53:02,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 82.28 MB 2025-02-15 04:53:02,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35435.43 MB 2025-02-15 04:53:02,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:53:02,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:53:02,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:53:02,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31560.58 MB 2025-02-15 04:53:02,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31853.38 MB 2025-02-15 04:53:02,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.81 MB 2025-02-15 04:53:02,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32073.09 MB 2025-02-15 04:53:02,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:53:02,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:53:02,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:53:02,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31853.38 MB 2025-02-15 04:53:02,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32209.84 MB 2025-02-15 04:53:02,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.45 MB 2025-02-15 04:53:02,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33061.02 MB 2025-02-15 04:53:02,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:53:02,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:53:02,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:53:02,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31560.58 MB 2025-02-15 04:53:02,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32209.84 MB 2025-02-15 04:53:02,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 649.26 MB 2025-02-15 04:53:02,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40930.12 MB 2025-02-15 04:53:02,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33061.02 MB 2025-02-15 04:53:02,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:53:02,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:53:02,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:53:02,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32553.90 MB 2025-02-15 04:53:02,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32703.26 MB 2025-02-15 04:53:02,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.36 MB 2025-02-15 04:53:02,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40930.12 MB 2025-02-15 04:53:02,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41024.49 MB 2025-02-15 04:53:02,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 94.37 MB 2025-02-15 04:53:02,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32812.97 MB 2025-02-15 04:53:02,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:53:02,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:53:02,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:53:02,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32797.74 MB 2025-02-15 04:53:02,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32947.72 MB 2025-02-15 04:53:02,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.98 MB 2025-02-15 04:53:02,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41024.49 MB 2025-02-15 04:53:02,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41024.49 MB 2025-02-15 04:53:02,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32947.72 MB 2025-02-15 04:53:02,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:53:02,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:53:02,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-15 04:53:02,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30730.87 MB 2025-02-15 04:53:02,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33081.63 MB 2025-02-15 04:53:02,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2350.75 MB 2025-02-15 04:53:02,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68526.54 MB 2025-02-15 04:53:02,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41024.49 MB 2025-02-15 04:53:02,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27502.05 MB 2025-02-15 04:53:02,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33081.63 MB 2025-02-15 04:53:02,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:53:02,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:53:02,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 04:53:02,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33081.63 MB 2025-02-15 04:53:02,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33099.32 MB 2025-02-15 04:53:02,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 17.69 MB 2025-02-15 04:53:02,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41024.49 MB 2025-02-15 04:53:02,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41024.49 MB 2025-02-15 04:53:02,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33750.71 MB 2025-02-15 04:53:02,712 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5431, cut from 5433 2025-02-15 04:53:02,712 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:53:02,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:53:02,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:53:02,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:53:02,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:02,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33099.32 MB 2025-02-15 04:53:02,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38719.36 MB 2025-02-15 04:53:02,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5620.04 MB 2025-02-15 04:53:02,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41024.49 MB 2025-02-15 04:53:02,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41024.49 MB 2025-02-15 04:53:02,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:02,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38719.36 MB 2025-02-15 04:53:02,825 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5223] 2025-02-15 04:53:02,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:02,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:53:02,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:02,828 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:53:02,832 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:53:02,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:02,834 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:53:02,834 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:53:17,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:17,699 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:53:17,704 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:53:17,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:17,707 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 237, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:53:17,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:17,708 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 237, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:53:21,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:53:21,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:53:21,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.67 seconds 2025-02-15 04:53:21,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:21,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31748.23 MB 2025-02-15 04:53:21,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32586.96 MB 2025-02-15 04:53:21,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 838.73 MB 2025-02-15 04:53:21,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49406.80 MB 2025-02-15 04:53:21,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43232.79 MB 2025-02-15 04:53:21,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6174.02 MB 2025-02-15 04:53:21,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41446.09 MB 2025-02-15 04:53:21,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:53:21,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:53:21,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:53:21,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:21,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32586.96 MB 2025-02-15 04:53:21,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32902.47 MB 2025-02-15 04:53:21,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.52 MB 2025-02-15 04:53:21,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43232.79 MB 2025-02-15 04:53:21,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43232.79 MB 2025-02-15 04:53:21,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:21,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35733.79 MB 2025-02-15 04:53:22,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:53:22,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:53:22,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.08 seconds 2025-02-15 04:53:22,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32902.47 MB 2025-02-15 04:53:22,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33199.74 MB 2025-02-15 04:53:22,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 297.27 MB 2025-02-15 04:53:22,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43232.79 MB 2025-02-15 04:53:22,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40458.26 MB 2025-02-15 04:53:22,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2774.53 MB 2025-02-15 04:53:22,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37157.06 MB 2025-02-15 04:53:22,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:53:22,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:53:22,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:53:22,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33199.74 MB 2025-02-15 04:53:22,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34257.62 MB 2025-02-15 04:53:22,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1057.88 MB 2025-02-15 04:53:22,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-15 04:53:22,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40458.26 MB 2025-02-15 04:53:22,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:22,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35051.39 MB 2025-02-15 04:53:22,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:53:22,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:53:22,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 04:53:22,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34257.62 MB 2025-02-15 04:53:22,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35513.09 MB 2025-02-15 04:53:22,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1255.47 MB 2025-02-15 04:53:22,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-15 04:53:22,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40458.26 MB 2025-02-15 04:53:22,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:22,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38617.86 MB 2025-02-15 04:53:22,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:53:22,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:53:22,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 04:53:22,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33199.74 MB 2025-02-15 04:53:22,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35513.09 MB 2025-02-15 04:53:22,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2313.35 MB 2025-02-15 04:53:22,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-15 04:53:22,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40458.26 MB 2025-02-15 04:53:22,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:22,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38617.86 MB 2025-02-15 04:53:22,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:53:22,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:53:22,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:53:22,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36371.88 MB 2025-02-15 04:53:22,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36801.40 MB 2025-02-15 04:53:22,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 429.52 MB 2025-02-15 04:53:22,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-15 04:53:22,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40693.14 MB 2025-02-15 04:53:22,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 234.88 MB 2025-02-15 04:53:22,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37198.00 MB 2025-02-15 04:53:22,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:53:22,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:53:22,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:53:22,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37032.62 MB 2025-02-15 04:53:22,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37261.41 MB 2025-02-15 04:53:22,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 04:53:22,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40693.14 MB 2025-02-15 04:53:22,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40693.14 MB 2025-02-15 04:53:22,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:22,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37313.83 MB 2025-02-15 04:53:22,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:53:22,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:53:22,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.01 seconds 2025-02-15 04:53:22,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30922.50 MB 2025-02-15 04:53:22,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37462.14 MB 2025-02-15 04:53:22,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6539.64 MB 2025-02-15 04:53:22,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49406.80 MB 2025-02-15 04:53:22,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40693.14 MB 2025-02-15 04:53:22,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8713.67 MB 2025-02-15 04:53:22,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37462.14 MB 2025-02-15 04:53:22,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:53:22,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:53:22,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:53:22,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:22,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32082.09 MB 2025-02-15 04:53:22,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35090.96 MB 2025-02-15 04:53:22,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.87 MB 2025-02-15 04:53:22,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40693.14 MB 2025-02-15 04:53:22,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40693.14 MB 2025-02-15 04:53:22,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:53:22,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35391.81 MB 2025-02-15 04:53:23,009 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 04:53:23,009 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:53:23,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:53:23,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:53:23,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:53:23,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:53:23,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35090.96 MB 2025-02-15 04:53:23,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43515.91 MB 2025-02-15 04:53:23,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 04:53:23,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40693.14 MB 2025-02-15 04:53:23,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49069.16 MB 2025-02-15 04:53:23,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 04:53:23,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43515.91 MB 2025-02-15 04:53:23,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 04:53:23,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:23,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:53:23,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:23,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:53:23,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:53:23,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:53:23,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:53:23,181 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:54:11,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:11,839 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:54:11,844 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:54:11,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:11,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:54:11,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:11,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:54:15,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:54:15,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:54:15,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.98 seconds 2025-02-15 04:54:15,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:15,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31887.59 MB 2025-02-15 04:54:15,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32797.10 MB 2025-02-15 04:54:15,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-15 04:54:15,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57445.19 MB 2025-02-15 04:54:15,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:54:15,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19524.49 MB 2025-02-15 04:54:15,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41811.94 MB 2025-02-15 04:54:15,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:54:15,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:54:15,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:54:15,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:15,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32797.10 MB 2025-02-15 04:54:15,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32823.33 MB 2025-02-15 04:54:15,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.23 MB 2025-02-15 04:54:15,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:54:15,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:54:15,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:54:15,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35578.21 MB 2025-02-15 04:54:16,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:54:16,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:54:16,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-15 04:54:16,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:16,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32823.33 MB 2025-02-15 04:54:16,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33086.09 MB 2025-02-15 04:54:16,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-15 04:54:16,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:54:16,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:54:16,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:54:16,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37077.91 MB 2025-02-15 04:54:16,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:54:16,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:54:16,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:54:16,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:16,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33086.09 MB 2025-02-15 04:54:16,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34021.19 MB 2025-02-15 04:54:16,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 935.09 MB 2025-02-15 04:54:16,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:54:16,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 04:54:16,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:54:16,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34722.82 MB 2025-02-15 04:54:16,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:54:16,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:54:16,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:54:16,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:16,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34021.19 MB 2025-02-15 04:54:16,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35130.94 MB 2025-02-15 04:54:16,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.75 MB 2025-02-15 04:54:16,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:54:16,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39558.58 MB 2025-02-15 04:54:16,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1637.88 MB 2025-02-15 04:54:16,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37876.90 MB 2025-02-15 04:54:16,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:54:16,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:54:16,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:54:16,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:16,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33086.09 MB 2025-02-15 04:54:16,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35130.94 MB 2025-02-15 04:54:16,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2044.84 MB 2025-02-15 04:54:16,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 04:54:16,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39558.58 MB 2025-02-15 04:54:16,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1637.88 MB 2025-02-15 04:54:16,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37876.90 MB 2025-02-15 04:54:17,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:54:17,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:54:17,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:54:17,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:17,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35890.04 MB 2025-02-15 04:54:17,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36269.71 MB 2025-02-15 04:54:17,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.67 MB 2025-02-15 04:54:17,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39558.58 MB 2025-02-15 04:54:17,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 04:54:17,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-15 04:54:17,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36624.87 MB 2025-02-15 04:54:17,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:54:17,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:54:17,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:54:17,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:17,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36474.09 MB 2025-02-15 04:54:17,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36693.69 MB 2025-02-15 04:54:17,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.59 MB 2025-02-15 04:54:17,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39766.20 MB 2025-02-15 04:54:17,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 04:54:17,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:54:17,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36742.90 MB 2025-02-15 04:54:17,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:54:17,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:54:17,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.16 seconds 2025-02-15 04:54:17,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:17,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30992.18 MB 2025-02-15 04:54:17,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36894.71 MB 2025-02-15 04:54:17,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5902.53 MB 2025-02-15 04:54:17,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57445.19 MB 2025-02-15 04:54:17,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 04:54:17,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17678.99 MB 2025-02-15 04:54:17,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36894.71 MB 2025-02-15 04:54:17,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:54:17,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:54:17,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:54:17,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:17,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32029.21 MB 2025-02-15 04:54:17,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35042.51 MB 2025-02-15 04:54:17,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3013.30 MB 2025-02-15 04:54:17,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39766.20 MB 2025-02-15 04:54:17,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 04:54:17,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:54:17,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35343.80 MB 2025-02-15 04:54:17,302 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 04:54:17,302 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:54:17,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:54:17,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:54:17,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:54:17,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:54:17,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35042.51 MB 2025-02-15 04:54:17,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43479.98 MB 2025-02-15 04:54:17,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 04:54:17,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39766.20 MB 2025-02-15 04:54:17,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48154.80 MB 2025-02-15 04:54:17,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 04:54:17,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43479.98 MB 2025-02-15 04:54:17,470 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 04:54:17,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:17,471 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:54:17,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:17,472 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:54:17,477 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:54:17,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:17,478 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:54:17,478 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:54:46,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:46,342 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:54:46,347 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:54:46,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:46,351 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:54:46,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:54:46,352 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:55:03,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:55:03,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:55:03,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.03 seconds 2025-02-15 04:55:03,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:03,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37810.53 MB 2025-02-15 04:55:03,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41728.14 MB 2025-02-15 04:55:03,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3917.61 MB 2025-02-15 04:55:03,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56543.41 MB 2025-02-15 04:55:03,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48119.15 MB 2025-02-15 04:55:03,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8424.26 MB 2025-02-15 04:55:03,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50679.28 MB 2025-02-15 04:55:03,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:55:03,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:55:03,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:55:03,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:03,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41728.14 MB 2025-02-15 04:55:03,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38660.86 MB 2025-02-15 04:55:03,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3067.27 MB 2025-02-15 04:55:03,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48119.15 MB 2025-02-15 04:55:03,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56314.82 MB 2025-02-15 04:55:03,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8195.67 MB 2025-02-15 04:55:03,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52730.20 MB 2025-02-15 04:55:05,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:55:05,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:55:05,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:55:05,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38660.86 MB 2025-02-15 04:55:05,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39191.71 MB 2025-02-15 04:55:05,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:55:05,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56314.82 MB 2025-02-15 04:55:05,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45615.15 MB 2025-02-15 04:55:05,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10699.67 MB 2025-02-15 04:55:05,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43170.25 MB 2025-02-15 04:55:05,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:55:05,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:55:05,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:05,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39191.71 MB 2025-02-15 04:55:05,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41081.24 MB 2025-02-15 04:55:05,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:55:05,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45615.15 MB 2025-02-15 04:55:05,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45615.15 MB 2025-02-15 04:55:05,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:05,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42498.67 MB 2025-02-15 04:55:05,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:55:05,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:55:05,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:55:05,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41081.24 MB 2025-02-15 04:55:05,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43323.10 MB 2025-02-15 04:55:05,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:55:05,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45615.15 MB 2025-02-15 04:55:05,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51277.46 MB 2025-02-15 04:55:05,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:55:05,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48867.38 MB 2025-02-15 04:55:05,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:55:05,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:55:05,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:55:05,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39191.71 MB 2025-02-15 04:55:05,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43323.10 MB 2025-02-15 04:55:05,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:55:05,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45615.15 MB 2025-02-15 04:55:05,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51277.46 MB 2025-02-15 04:55:05,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:55:05,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48867.38 MB 2025-02-15 04:55:05,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:55:05,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:55:05,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 04:55:05,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44856.64 MB 2025-02-15 04:55:05,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45623.64 MB 2025-02-15 04:55:05,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:55:05,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51277.46 MB 2025-02-15 04:55:05,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 04:55:05,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:55:05,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46331.43 MB 2025-02-15 04:55:05,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:55:05,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:55:05,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:05,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46036.53 MB 2025-02-15 04:55:05,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46265.98 MB 2025-02-15 04:55:05,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.45 MB 2025-02-15 04:55:05,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51696.89 MB 2025-02-15 04:55:05,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 04:55:05,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:05,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46453.46 MB 2025-02-15 04:55:05,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:55:05,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:55:05,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.45 seconds 2025-02-15 04:55:05,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:05,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33953.65 MB 2025-02-15 04:55:05,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46467.05 MB 2025-02-15 04:55:05,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12513.41 MB 2025-02-15 04:55:05,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56543.41 MB 2025-02-15 04:55:05,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 04:55:05,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4846.52 MB 2025-02-15 04:55:05,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46467.05 MB 2025-02-15 04:55:06,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:55:06,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:55:06,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:55:06,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:06,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46467.05 MB 2025-02-15 04:55:06,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38958.04 MB 2025-02-15 04:55:06,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7509.02 MB 2025-02-15 04:55:06,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51696.89 MB 2025-02-15 04:55:06,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 04:55:06,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:06,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48978.72 MB 2025-02-15 04:55:06,085 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:55:06,085 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:55:06,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:55:06,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:55:06,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:06,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:06,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38958.04 MB 2025-02-15 04:55:06,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47397.06 MB 2025-02-15 04:55:06,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:55:06,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51696.89 MB 2025-02-15 04:55:06,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60087.60 MB 2025-02-15 04:55:06,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 04:55:06,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47397.06 MB 2025-02-15 04:55:06,253 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:55:06,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:06,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:55:06,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:06,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:55:06,260 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:55:06,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:06,261 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:55:06,261 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:55:17,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:17,580 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:55:17,585 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:55:17,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:17,589 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 770, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:55:17,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:17,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 770, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:55:29,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:55:29,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:55:29,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.01 seconds 2025-02-15 04:55:29,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:29,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35462.26 MB 2025-02-15 04:55:29,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38187.24 MB 2025-02-15 04:55:29,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2724.99 MB 2025-02-15 04:55:29,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72672.61 MB 2025-02-15 04:55:29,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44147.15 MB 2025-02-15 04:55:29,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28525.46 MB 2025-02-15 04:55:29,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47198.55 MB 2025-02-15 04:55:29,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:55:29,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:55:29,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 04:55:29,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:29,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38187.24 MB 2025-02-15 04:55:29,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36908.91 MB 2025-02-15 04:55:29,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1278.34 MB 2025-02-15 04:55:29,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44147.15 MB 2025-02-15 04:55:29,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50354.72 MB 2025-02-15 04:55:29,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6207.57 MB 2025-02-15 04:55:29,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47202.84 MB 2025-02-15 04:55:31,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:55:31,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:55:31,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 04:55:31,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:31,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36908.91 MB 2025-02-15 04:55:31,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37439.75 MB 2025-02-15 04:55:31,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:55:31,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50354.72 MB 2025-02-15 04:55:31,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42836.43 MB 2025-02-15 04:55:31,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7518.29 MB 2025-02-15 04:55:31,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41418.29 MB 2025-02-15 04:55:31,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:55:31,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:55:31,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:31,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:31,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37439.75 MB 2025-02-15 04:55:31,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39329.28 MB 2025-02-15 04:55:31,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:55:31,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42836.43 MB 2025-02-15 04:55:31,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43780.15 MB 2025-02-15 04:55:31,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:55:31,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40746.71 MB 2025-02-15 04:55:31,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:55:31,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:55:31,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:55:31,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:31,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39329.28 MB 2025-02-15 04:55:31,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41571.14 MB 2025-02-15 04:55:31,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:55:31,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43780.15 MB 2025-02-15 04:55:31,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49442.46 MB 2025-02-15 04:55:31,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:55:31,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47115.42 MB 2025-02-15 04:55:31,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:55:31,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:55:31,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:55:31,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:31,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37439.75 MB 2025-02-15 04:55:31,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41571.14 MB 2025-02-15 04:55:31,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:55:31,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42836.43 MB 2025-02-15 04:55:31,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49442.46 MB 2025-02-15 04:55:31,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:55:31,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47115.42 MB 2025-02-15 04:55:32,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:55:32,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:55:32,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:55:32,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:32,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43104.68 MB 2025-02-15 04:55:32,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43871.68 MB 2025-02-15 04:55:32,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:55:32,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49442.46 MB 2025-02-15 04:55:32,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49861.89 MB 2025-02-15 04:55:32,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:55:32,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44579.47 MB 2025-02-15 04:55:32,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:55:32,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:55:32,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:32,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:32,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44284.57 MB 2025-02-15 04:55:32,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44511.54 MB 2025-02-15 04:55:32,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.97 MB 2025-02-15 04:55:32,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49861.89 MB 2025-02-15 04:55:32,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49861.89 MB 2025-02-15 04:55:32,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:32,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44702.39 MB 2025-02-15 04:55:32,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:55:32,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:55:32,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.43 seconds 2025-02-15 04:55:32,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:32,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32779.51 MB 2025-02-15 04:55:32,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44712.19 MB 2025-02-15 04:55:32,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11932.68 MB 2025-02-15 04:55:32,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72672.61 MB 2025-02-15 04:55:32,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49861.89 MB 2025-02-15 04:55:32,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22810.72 MB 2025-02-15 04:55:32,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44712.19 MB 2025-02-15 04:55:32,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:55:32,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:55:32,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:55:32,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:32,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44712.19 MB 2025-02-15 04:55:32,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37776.82 MB 2025-02-15 04:55:32,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6935.37 MB 2025-02-15 04:55:32,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49861.89 MB 2025-02-15 04:55:32,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49861.89 MB 2025-02-15 04:55:32,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:32,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47218.64 MB 2025-02-15 04:55:32,310 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 04:55:32,311 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:55:32,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:55:32,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:55:32,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:32,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:32,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37776.82 MB 2025-02-15 04:55:32,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46198.78 MB 2025-02-15 04:55:32,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-15 04:55:32,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49861.89 MB 2025-02-15 04:55:32,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58233.72 MB 2025-02-15 04:55:32,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 04:55:32,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46198.78 MB 2025-02-15 04:55:32,477 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 04:55:32,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:32,478 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:55:32,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:32,479 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:55:32,484 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:55:32,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:32,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:55:32,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:55:42,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:42,894 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:55:42,899 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:55:42,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:42,902 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 233, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:55:42,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:42,903 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 233, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:55:46,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:55:46,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:55:46,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.67 seconds 2025-02-15 04:55:46,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31720.94 MB 2025-02-15 04:55:46,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32545.52 MB 2025-02-15 04:55:46,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 824.57 MB 2025-02-15 04:55:46,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66605.55 MB 2025-02-15 04:55:46,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28907.14 MB 2025-02-15 04:55:46,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41418.81 MB 2025-02-15 04:55:46,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:55:46,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:55:46,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:46,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32545.52 MB 2025-02-15 04:55:46,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31414.00 MB 2025-02-15 04:55:46,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1131.52 MB 2025-02-15 04:55:46,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32762.13 MB 2025-02-15 04:55:46,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:55:46,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:55:46,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:55:46,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31414.00 MB 2025-02-15 04:55:46,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31433.91 MB 2025-02-15 04:55:46,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.91 MB 2025-02-15 04:55:46,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32371.36 MB 2025-02-15 04:55:46,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:55:46,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:55:46,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:55:46,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31433.84 MB 2025-02-15 04:55:46,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31504.68 MB 2025-02-15 04:55:46,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 70.84 MB 2025-02-15 04:55:46,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31558.50 MB 2025-02-15 04:55:46,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:55:46,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:55:46,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:46,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31504.68 MB 2025-02-15 04:55:46,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31589.40 MB 2025-02-15 04:55:46,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 84.71 MB 2025-02-15 04:55:46,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31799.51 MB 2025-02-15 04:55:46,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:55:46,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:55:46,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:46,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31433.84 MB 2025-02-15 04:55:46,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31589.40 MB 2025-02-15 04:55:46,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 155.55 MB 2025-02-15 04:55:46,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:46,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31799.51 MB 2025-02-15 04:55:46,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:55:46,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:55:46,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:46,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31648.38 MB 2025-02-15 04:55:46,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31677.44 MB 2025-02-15 04:55:46,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 29.06 MB 2025-02-15 04:55:46,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:46,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37713.08 MB 2025-02-15 04:55:46,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14.68 MB 2025-02-15 04:55:46,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31713.92 MB 2025-02-15 04:55:46,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:55:46,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:55:46,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:55:46,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31692.93 MB 2025-02-15 04:55:46,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31712.78 MB 2025-02-15 04:55:46,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.85 MB 2025-02-15 04:55:46,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37713.08 MB 2025-02-15 04:55:46,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37713.08 MB 2025-02-15 04:55:46,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31712.78 MB 2025-02-15 04:55:46,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:55:46,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:55:46,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-15 04:55:46,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30909.15 MB 2025-02-15 04:55:46,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31749.59 MB 2025-02-15 04:55:46,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.44 MB 2025-02-15 04:55:46,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66605.55 MB 2025-02-15 04:55:46,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37713.08 MB 2025-02-15 04:55:46,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28892.46 MB 2025-02-15 04:55:46,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31749.59 MB 2025-02-15 04:55:46,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:55:46,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:55:46,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:55:46,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31749.59 MB 2025-02-15 04:55:46,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32301.45 MB 2025-02-15 04:55:46,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 551.86 MB 2025-02-15 04:55:46,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37713.08 MB 2025-02-15 04:55:46,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37715.18 MB 2025-02-15 04:55:46,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 04:55:46,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32356.63 MB 2025-02-15 04:55:46,772 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1483, cut from 1485 2025-02-15 04:55:46,772 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:55:46,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:55:46,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:55:46,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:55:46,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:46,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31553.41 MB 2025-02-15 04:55:46,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33098.09 MB 2025-02-15 04:55:46,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1544.68 MB 2025-02-15 04:55:46,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37715.18 MB 2025-02-15 04:55:46,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37715.18 MB 2025-02-15 04:55:46,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:46,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33098.09 MB 2025-02-15 04:55:46,804 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1275] 2025-02-15 04:55:46,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:46,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:55:46,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:46,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:55:46,811 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:55:46,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:46,812 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:55:46,812 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 04:55:51,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:51,925 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:55:51,930 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:55:51,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:51,934 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:55:51,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:51,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:55:55,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:55:55,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:55:55,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.51 seconds 2025-02-15 04:55:55,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:55,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33447.32 MB 2025-02-15 04:55:55,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34243.58 MB 2025-02-15 04:55:55,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 796.26 MB 2025-02-15 04:55:55,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37715.18 MB 2025-02-15 04:55:55,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-15 04:55:55,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16.78 MB 2025-02-15 04:55:55,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43145.18 MB 2025-02-15 04:55:55,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:55:55,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:55:55,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:55,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:55,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32460.80 MB 2025-02-15 04:55:55,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32663.99 MB 2025-02-15 04:55:55,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.19 MB 2025-02-15 04:55:55,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-15 04:55:55,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 04:55:55,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 04:55:55,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35256.03 MB 2025-02-15 04:55:56,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:55:56,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:55:56,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.99 seconds 2025-02-15 04:55:56,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32663.99 MB 2025-02-15 04:55:56,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32928.09 MB 2025-02-15 04:55:56,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.09 MB 2025-02-15 04:55:56,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 04:55:56,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 04:55:56,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:56,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36919.62 MB 2025-02-15 04:55:56,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:55:56,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:55:56,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:56,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32928.09 MB 2025-02-15 04:55:56,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33867.90 MB 2025-02-15 04:55:56,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 939.81 MB 2025-02-15 04:55:56,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 04:55:56,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 04:55:56,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:56,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34573.08 MB 2025-02-15 04:55:56,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:55:56,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:55:56,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:55:56,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33867.90 MB 2025-02-15 04:55:56,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34983.26 MB 2025-02-15 04:55:56,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1115.36 MB 2025-02-15 04:55:56,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 04:55:56,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39575.36 MB 2025-02-15 04:55:56,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2348.81 MB 2025-02-15 04:55:56,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37742.55 MB 2025-02-15 04:55:56,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:55:56,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:55:56,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:55:56,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32928.09 MB 2025-02-15 04:55:56,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34983.26 MB 2025-02-15 04:55:56,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2055.17 MB 2025-02-15 04:55:56,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 04:55:56,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39575.36 MB 2025-02-15 04:55:56,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2348.81 MB 2025-02-15 04:55:56,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37742.55 MB 2025-02-15 04:55:56,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:55:56,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:55:56,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:55:56,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35746.19 MB 2025-02-15 04:55:56,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36127.78 MB 2025-02-15 04:55:56,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 381.58 MB 2025-02-15 04:55:56,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39575.36 MB 2025-02-15 04:55:56,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 04:55:56,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-15 04:55:56,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36485.73 MB 2025-02-15 04:55:56,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:55:56,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:55:56,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:55:56,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36333.20 MB 2025-02-15 04:55:56,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36557.97 MB 2025-02-15 04:55:56,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.78 MB 2025-02-15 04:55:56,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 04:55:56,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 04:55:56,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:56,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36603.26 MB 2025-02-15 04:55:56,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:55:56,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:55:56,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.74 seconds 2025-02-15 04:55:56,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32663.40 MB 2025-02-15 04:55:56,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36759.05 MB 2025-02-15 04:55:56,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4095.65 MB 2025-02-15 04:55:56,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37715.18 MB 2025-02-15 04:55:56,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 04:55:56,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2067.79 MB 2025-02-15 04:55:56,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36759.05 MB 2025-02-15 04:55:56,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:55:56,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:55:56,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:55:56,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31922.23 MB 2025-02-15 04:55:56,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34936.27 MB 2025-02-15 04:55:56,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 04:55:56,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 04:55:56,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 04:55:56,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:55:56,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35237.63 MB 2025-02-15 04:55:56,967 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 04:55:56,967 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:55:56,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:55:56,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:55:56,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:55:56,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:55:56,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34936.27 MB 2025-02-15 04:55:56,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43375.29 MB 2025-02-15 04:55:56,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 04:55:56,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 04:55:56,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50272.93 MB 2025-02-15 04:55:56,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 04:55:56,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43375.29 MB 2025-02-15 04:55:57,135 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 04:55:57,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:57,136 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:55:57,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:57,137 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:55:57,142 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:55:57,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:55:57,143 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:55:57,143 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:56:06,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:06,200 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:56:06,208 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:56:06,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:06,214 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 88, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:56:06,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:06,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 88, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:56:07,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:56:07,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:56:07,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.43 seconds 2025-02-15 04:56:07,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:07,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30709.97 MB 2025-02-15 04:56:07,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31021.40 MB 2025-02-15 04:56:07,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.43 MB 2025-02-15 04:56:07,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62857.94 MB 2025-02-15 04:56:07,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:07,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26107.45 MB 2025-02-15 04:56:07,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39954.85 MB 2025-02-15 04:56:07,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:56:07,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:56:07,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:56:07,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:07,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31021.40 MB 2025-02-15 04:56:07,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31172.28 MB 2025-02-15 04:56:07,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 150.89 MB 2025-02-15 04:56:07,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:07,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:07,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:07,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31639.49 MB 2025-02-15 04:56:08,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:56:08,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:56:08,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.45 seconds 2025-02-15 04:56:08,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31172.28 MB 2025-02-15 04:56:08,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31289.00 MB 2025-02-15 04:56:08,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 116.72 MB 2025-02-15 04:56:08,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:08,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:08,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35256.93 MB 2025-02-15 04:56:08,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:56:08,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:56:08,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:56:08,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31289.00 MB 2025-02-15 04:56:08,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31704.60 MB 2025-02-15 04:56:08,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 415.60 MB 2025-02-15 04:56:08,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:08,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:08,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32016.44 MB 2025-02-15 04:56:08,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:56:08,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:56:08,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 04:56:08,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31704.60 MB 2025-02-15 04:56:08,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32210.13 MB 2025-02-15 04:56:08,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 505.53 MB 2025-02-15 04:56:08,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:08,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:08,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33417.55 MB 2025-02-15 04:56:08,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:56:08,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:56:08,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 04:56:08,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31289.00 MB 2025-02-15 04:56:08,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32210.13 MB 2025-02-15 04:56:08,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 921.13 MB 2025-02-15 04:56:08,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:08,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:56:08,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33417.55 MB 2025-02-15 04:56:08,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:56:08,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:56:08,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 04:56:08,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32697.72 MB 2025-02-15 04:56:08,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32909.72 MB 2025-02-15 04:56:08,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.99 MB 2025-02-15 04:56:08,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:56:08,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36886.81 MB 2025-02-15 04:56:08,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-15 04:56:08,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33065.43 MB 2025-02-15 04:56:08,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:56:08,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:56:08,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:56:08,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33043.82 MB 2025-02-15 04:56:08,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33254.29 MB 2025-02-15 04:56:08,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.48 MB 2025-02-15 04:56:08,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36886.81 MB 2025-02-15 04:56:08,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36886.81 MB 2025-02-15 04:56:08,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33254.29 MB 2025-02-15 04:56:08,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:56:08,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:56:08,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.09 seconds 2025-02-15 04:56:08,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30403.37 MB 2025-02-15 04:56:08,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33441.91 MB 2025-02-15 04:56:08,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3038.54 MB 2025-02-15 04:56:08,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62857.94 MB 2025-02-15 04:56:08,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36886.81 MB 2025-02-15 04:56:08,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25971.13 MB 2025-02-15 04:56:08,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33441.91 MB 2025-02-15 04:56:08,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:56:08,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:56:08,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:56:08,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30914.57 MB 2025-02-15 04:56:08,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33726.95 MB 2025-02-15 04:56:08,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2812.39 MB 2025-02-15 04:56:08,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36886.81 MB 2025-02-15 04:56:08,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36886.81 MB 2025-02-15 04:56:08,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:08,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34008.16 MB 2025-02-15 04:56:08,605 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7615, cut from 7617 2025-02-15 04:56:08,606 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:56:08,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:56:08,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:56:08,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:56:08,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:08,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33726.95 MB 2025-02-15 04:56:08,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41601.03 MB 2025-02-15 04:56:08,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.08 MB 2025-02-15 04:56:08,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36886.81 MB 2025-02-15 04:56:08,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44717.57 MB 2025-02-15 04:56:08,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7830.77 MB 2025-02-15 04:56:08,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41601.03 MB 2025-02-15 04:56:08,852 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7407] 2025-02-15 04:56:08,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:08,855 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:56:08,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:08,857 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:56:08,864 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:56:08,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:08,867 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:56:08,867 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 04:56:13,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:13,849 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:56:13,857 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:56:13,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:13,863 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:56:13,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:13,865 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:56:16,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:56:16,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:56:16,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-15 04:56:16,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:16,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35972.42 MB 2025-02-15 04:56:16,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36535.11 MB 2025-02-15 04:56:16,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-15 04:56:16,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52548.34 MB 2025-02-15 04:56:16,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 04:56:16,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11878.27 MB 2025-02-15 04:56:16,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45443.79 MB 2025-02-15 04:56:16,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:56:16,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:56:16,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:56:16,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:16,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36535.11 MB 2025-02-15 04:56:16,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36807.67 MB 2025-02-15 04:56:16,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.56 MB 2025-02-15 04:56:16,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 04:56:16,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 04:56:16,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:16,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38768.43 MB 2025-02-15 04:56:17,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:56:17,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:56:17,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 04:56:17,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36807.67 MB 2025-02-15 04:56:17,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37018.68 MB 2025-02-15 04:56:17,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.01 MB 2025-02-15 04:56:17,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 04:56:17,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 04:56:17,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:17,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40978.36 MB 2025-02-15 04:56:17,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:56:17,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:56:17,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:56:17,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37018.68 MB 2025-02-15 04:56:17,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37769.59 MB 2025-02-15 04:56:17,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.91 MB 2025-02-15 04:56:17,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 04:56:17,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 04:56:17,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:17,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38333.02 MB 2025-02-15 04:56:17,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:56:17,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:56:17,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:56:17,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37769.59 MB 2025-02-15 04:56:17,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38660.76 MB 2025-02-15 04:56:17,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 891.18 MB 2025-02-15 04:56:17,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 04:56:17,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42360.37 MB 2025-02-15 04:56:17,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1690.30 MB 2025-02-15 04:56:17,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40865.63 MB 2025-02-15 04:56:17,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:56:17,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:56:17,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 04:56:17,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37018.68 MB 2025-02-15 04:56:17,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38660.76 MB 2025-02-15 04:56:17,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.08 MB 2025-02-15 04:56:17,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 04:56:17,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42360.37 MB 2025-02-15 04:56:17,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1690.30 MB 2025-02-15 04:56:17,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40865.63 MB 2025-02-15 04:56:17,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:56:17,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:56:17,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:56:17,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39270.35 MB 2025-02-15 04:56:17,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39575.23 MB 2025-02-15 04:56:17,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.88 MB 2025-02-15 04:56:17,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42360.37 MB 2025-02-15 04:56:17,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42523.95 MB 2025-02-15 04:56:17,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 04:56:17,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39863.46 MB 2025-02-15 04:56:17,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:56:17,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:56:17,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:56:17,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39739.36 MB 2025-02-15 04:56:17,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39959.62 MB 2025-02-15 04:56:17,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.26 MB 2025-02-15 04:56:17,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42523.95 MB 2025-02-15 04:56:17,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42523.95 MB 2025-02-15 04:56:17,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:17,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39978.11 MB 2025-02-15 04:56:17,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:56:17,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:56:17,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-15 04:56:17,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35418.45 MB 2025-02-15 04:56:17,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40160.48 MB 2025-02-15 04:56:17,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.02 MB 2025-02-15 04:56:17,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52548.34 MB 2025-02-15 04:56:17,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42523.95 MB 2025-02-15 04:56:17,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10024.39 MB 2025-02-15 04:56:17,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40160.48 MB 2025-02-15 04:56:17,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:56:17,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:56:17,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:56:17,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40160.48 MB 2025-02-15 04:56:17,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39281.88 MB 2025-02-15 04:56:17,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -878.59 MB 2025-02-15 04:56:17,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42523.95 MB 2025-02-15 04:56:17,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42523.95 MB 2025-02-15 04:56:17,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:56:17,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40963.32 MB 2025-02-15 04:56:17,648 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 04:56:17,649 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 04:56:17,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:56:17,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:56:17,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:56:17,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:56:17,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39281.88 MB 2025-02-15 04:56:17,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47712.28 MB 2025-02-15 04:56:17,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 04:56:17,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42523.95 MB 2025-02-15 04:56:17,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52999.23 MB 2025-02-15 04:56:17,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 04:56:17,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47712.28 MB 2025-02-15 04:56:17,815 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 04:56:17,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:17,816 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:56:17,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:17,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:56:17,822 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:56:17,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:56:17,823 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:56:17,823 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 04:57:24,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:24,755 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:57:24,760 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:57:24,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:24,763 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 61, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:57:24,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:24,764 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 61, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:57:25,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:57:25,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:57:25,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 04:57:25,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:25,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30521.83 MB 2025-02-15 04:57:25,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30737.71 MB 2025-02-15 04:57:25,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 215.88 MB 2025-02-15 04:57:25,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61379.44 MB 2025-02-15 04:57:25,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36750.49 MB 2025-02-15 04:57:25,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24628.95 MB 2025-02-15 04:57:25,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39333.36 MB 2025-02-15 04:57:25,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:57:25,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:57:25,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:57:25,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:25,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30737.71 MB 2025-02-15 04:57:25,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30842.30 MB 2025-02-15 04:57:25,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 104.59 MB 2025-02-15 04:57:25,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36750.49 MB 2025-02-15 04:57:25,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36375.10 MB 2025-02-15 04:57:25,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -375.39 MB 2025-02-15 04:57:25,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31166.17 MB 2025-02-15 04:57:26,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:57:26,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:57:26,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 04:57:26,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30842.30 MB 2025-02-15 04:57:26,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30923.25 MB 2025-02-15 04:57:26,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-15 04:57:26,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36375.10 MB 2025-02-15 04:57:26,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36375.10 MB 2025-02-15 04:57:26,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34735.55 MB 2025-02-15 04:57:26,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:57:26,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:57:26,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:57:26,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30923.18 MB 2025-02-15 04:57:26,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31211.27 MB 2025-02-15 04:57:26,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-15 04:57:26,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36375.10 MB 2025-02-15 04:57:26,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36375.10 MB 2025-02-15 04:57:26,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31427.43 MB 2025-02-15 04:57:26,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:57:26,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:57:26,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:57:26,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31211.27 MB 2025-02-15 04:57:26,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31561.20 MB 2025-02-15 04:57:26,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-15 04:57:26,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36375.10 MB 2025-02-15 04:57:26,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36375.10 MB 2025-02-15 04:57:26,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32398.65 MB 2025-02-15 04:57:26,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:57:26,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:57:26,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 04:57:26,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30923.25 MB 2025-02-15 04:57:26,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31561.20 MB 2025-02-15 04:57:26,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.95 MB 2025-02-15 04:57:26,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36375.10 MB 2025-02-15 04:57:26,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36375.10 MB 2025-02-15 04:57:26,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32398.65 MB 2025-02-15 04:57:26,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:57:26,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:57:26,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 04:57:26,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31899.91 MB 2025-02-15 04:57:26,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32047.35 MB 2025-02-15 04:57:26,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.44 MB 2025-02-15 04:57:26,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36375.10 MB 2025-02-15 04:57:26,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-15 04:57:26,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 94.37 MB 2025-02-15 04:57:26,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32155.29 MB 2025-02-15 04:57:26,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:57:26,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:57:26,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 04:57:26,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32140.31 MB 2025-02-15 04:57:26,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32287.06 MB 2025-02-15 04:57:26,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.75 MB 2025-02-15 04:57:26,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-15 04:57:26,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-15 04:57:26,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32287.06 MB 2025-02-15 04:57:26,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:57:26,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:57:26,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.35 seconds 2025-02-15 04:57:26,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30309.30 MB 2025-02-15 04:57:26,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32418.90 MB 2025-02-15 04:57:26,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2109.60 MB 2025-02-15 04:57:26,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61379.44 MB 2025-02-15 04:57:26,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-15 04:57:26,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24909.97 MB 2025-02-15 04:57:26,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32418.90 MB 2025-02-15 04:57:26,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:57:26,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:57:26,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 04:57:26,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32418.90 MB 2025-02-15 04:57:26,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32641.61 MB 2025-02-15 04:57:26,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.70 MB 2025-02-15 04:57:26,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-15 04:57:26,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-15 04:57:26,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:57:26,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34395.18 MB 2025-02-15 04:57:26,295 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-15 04:57:26,295 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:57:26,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:57:26,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:57:26,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:57:26,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:57:26,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32641.61 MB 2025-02-15 04:57:26,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38175.46 MB 2025-02-15 04:57:26,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.85 MB 2025-02-15 04:57:26,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-15 04:57:26,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41972.40 MB 2025-02-15 04:57:26,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5502.93 MB 2025-02-15 04:57:26,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38175.46 MB 2025-02-15 04:57:26,405 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-15 04:57:26,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:26,407 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:57:26,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:26,408 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:57:26,412 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:57:26,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:57:26,414 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:57:26,414 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:58:10,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:10,581 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:58:10,589 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:58:10,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:10,596 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1504, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:58:10,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:10,598 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1504, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:58:33,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:58:33,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:58:33,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.13 seconds 2025-02-15 04:58:33,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:33,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40576.88 MB 2025-02-15 04:58:33,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45899.46 MB 2025-02-15 04:58:33,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5322.57 MB 2025-02-15 04:58:33,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47475.33 MB 2025-02-15 04:58:33,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52139.39 MB 2025-02-15 04:58:33,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4664.07 MB 2025-02-15 04:58:33,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54804.60 MB 2025-02-15 04:58:33,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:58:33,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:58:33,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 04:58:33,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:33,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45899.46 MB 2025-02-15 04:58:33,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40724.74 MB 2025-02-15 04:58:33,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5174.71 MB 2025-02-15 04:58:33,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52139.39 MB 2025-02-15 04:58:33,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64015.56 MB 2025-02-15 04:58:33,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11876.17 MB 2025-02-15 04:58:33,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61627.09 MB 2025-02-15 04:58:35,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:58:35,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:58:35,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 04:58:35,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:35,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40724.74 MB 2025-02-15 04:58:35,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41255.58 MB 2025-02-15 04:58:35,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:58:35,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64015.56 MB 2025-02-15 04:58:35,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45480.94 MB 2025-02-15 04:58:35,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18534.63 MB 2025-02-15 04:58:35,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45235.17 MB 2025-02-15 04:58:35,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:58:35,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:58:35,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:58:35,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:35,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41255.58 MB 2025-02-15 04:58:35,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43145.12 MB 2025-02-15 04:58:35,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:58:35,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45480.94 MB 2025-02-15 04:58:35,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46424.65 MB 2025-02-15 04:58:35,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:58:35,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44562.55 MB 2025-02-15 04:58:36,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:58:36,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:58:36,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:58:36,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43145.12 MB 2025-02-15 04:58:36,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45386.97 MB 2025-02-15 04:58:36,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:58:36,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46424.65 MB 2025-02-15 04:58:36,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53030.68 MB 2025-02-15 04:58:36,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:58:36,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50931.26 MB 2025-02-15 04:58:36,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:58:36,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:58:36,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 04:58:36,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41255.58 MB 2025-02-15 04:58:36,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45386.97 MB 2025-02-15 04:58:36,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:58:36,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45480.94 MB 2025-02-15 04:58:36,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53030.68 MB 2025-02-15 04:58:36,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 04:58:36,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50931.26 MB 2025-02-15 04:58:36,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:58:36,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:58:36,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:58:36,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46920.52 MB 2025-02-15 04:58:36,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47687.52 MB 2025-02-15 04:58:36,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:58:36,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53030.68 MB 2025-02-15 04:58:36,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53450.11 MB 2025-02-15 04:58:36,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:58:36,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48395.31 MB 2025-02-15 04:58:36,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:58:36,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:58:36,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:58:36,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48100.41 MB 2025-02-15 04:58:36,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48328.73 MB 2025-02-15 04:58:36,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-15 04:58:36,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-15 04:58:36,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53450.11 MB 2025-02-15 04:58:36,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:58:36,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48552.70 MB 2025-02-15 04:58:36,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:58:36,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:58:36,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.60 seconds 2025-02-15 04:58:36,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35336.83 MB 2025-02-15 04:58:36,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48529.33 MB 2025-02-15 04:58:36,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13192.51 MB 2025-02-15 04:58:36,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47475.33 MB 2025-02-15 04:58:36,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53450.11 MB 2025-02-15 04:58:36,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5974.79 MB 2025-02-15 04:58:36,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48552.70 MB 2025-02-15 04:58:36,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:58:36,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:58:36,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:58:36,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48529.33 MB 2025-02-15 04:58:36,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40333.98 MB 2025-02-15 04:58:36,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8195.36 MB 2025-02-15 04:58:36,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-15 04:58:36,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53450.11 MB 2025-02-15 04:58:36,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:58:36,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51035.17 MB 2025-02-15 04:58:36,486 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 04:58:36,487 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:58:36,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:58:36,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:58:36,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:58:36,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:58:36,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40333.98 MB 2025-02-15 04:58:36,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48753.06 MB 2025-02-15 04:58:36,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 04:58:36,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-15 04:58:36,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61821.94 MB 2025-02-15 04:58:36,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 04:58:36,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48753.06 MB 2025-02-15 04:58:36,654 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 04:58:36,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:36,655 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:58:36,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:36,656 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:58:36,661 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:58:36,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:58:36,662 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:58:36,662 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 04:59:19,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:19,467 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 04:59:19,472 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 04:59:19,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:19,475 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 04:59:19,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:19,476 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 04:59:37,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 04:59:37,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 04:59:37,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.43 seconds 2025-02-15 04:59:37,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:37,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38423.72 MB 2025-02-15 04:59:37,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42653.68 MB 2025-02-15 04:59:37,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4229.96 MB 2025-02-15 04:59:37,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70193.77 MB 2025-02-15 04:59:37,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49729.77 MB 2025-02-15 04:59:37,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20464.01 MB 2025-02-15 04:59:37,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51518.97 MB 2025-02-15 04:59:37,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 04:59:37,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 04:59:37,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 04:59:37,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:37,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42653.68 MB 2025-02-15 04:59:37,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39118.35 MB 2025-02-15 04:59:37,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3535.33 MB 2025-02-15 04:59:37,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49729.77 MB 2025-02-15 04:59:37,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58126.76 MB 2025-02-15 04:59:37,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8397.00 MB 2025-02-15 04:59:37,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55277.10 MB 2025-02-15 04:59:39,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 04:59:39,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 04:59:39,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 04:59:39,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:39,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39118.35 MB 2025-02-15 04:59:39,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39649.19 MB 2025-02-15 04:59:39,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 04:59:39,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58126.76 MB 2025-02-15 04:59:39,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45499.81 MB 2025-02-15 04:59:39,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12626.95 MB 2025-02-15 04:59:39,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43627.74 MB 2025-02-15 04:59:39,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 04:59:39,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 04:59:39,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 04:59:39,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:39,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39649.19 MB 2025-02-15 04:59:39,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41538.72 MB 2025-02-15 04:59:39,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 04:59:39,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45499.81 MB 2025-02-15 04:59:39,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46443.53 MB 2025-02-15 04:59:39,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 04:59:39,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42956.15 MB 2025-02-15 04:59:40,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 04:59:40,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 04:59:40,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 04:59:40,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41538.72 MB 2025-02-15 04:59:40,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43780.58 MB 2025-02-15 04:59:40,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 04:59:40,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46443.53 MB 2025-02-15 04:59:40,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52105.84 MB 2025-02-15 04:59:40,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 04:59:40,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49324.86 MB 2025-02-15 04:59:40,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 04:59:40,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 04:59:40,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 04:59:40,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39649.19 MB 2025-02-15 04:59:40,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43780.58 MB 2025-02-15 04:59:40,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 04:59:40,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45499.81 MB 2025-02-15 04:59:40,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52105.84 MB 2025-02-15 04:59:40,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 04:59:40,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49324.86 MB 2025-02-15 04:59:40,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 04:59:40,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 04:59:40,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 04:59:40,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45314.12 MB 2025-02-15 04:59:40,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46081.12 MB 2025-02-15 04:59:40,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 04:59:40,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52105.84 MB 2025-02-15 04:59:40,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52525.27 MB 2025-02-15 04:59:40,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 04:59:40,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46788.91 MB 2025-02-15 04:59:40,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 04:59:40,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 04:59:40,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:59:40,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46494.01 MB 2025-02-15 04:59:40,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46722.02 MB 2025-02-15 04:59:40,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-15 04:59:40,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52525.27 MB 2025-02-15 04:59:40,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52525.27 MB 2025-02-15 04:59:40,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:59:40,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46946.99 MB 2025-02-15 04:59:40,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 04:59:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 04:59:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.86 seconds 2025-02-15 04:59:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34260.25 MB 2025-02-15 04:59:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46921.93 MB 2025-02-15 04:59:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12661.69 MB 2025-02-15 04:59:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70193.77 MB 2025-02-15 04:59:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52525.27 MB 2025-02-15 04:59:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17668.51 MB 2025-02-15 04:59:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46946.99 MB 2025-02-15 04:59:40,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 04:59:40,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 04:59:40,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 04:59:40,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46921.93 MB 2025-02-15 04:59:40,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39246.73 MB 2025-02-15 04:59:40,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7675.20 MB 2025-02-15 04:59:40,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52525.27 MB 2025-02-15 04:59:40,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52525.27 MB 2025-02-15 04:59:40,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 04:59:40,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49419.16 MB 2025-02-15 04:59:40,628 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 04:59:40,628 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 04:59:40,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 04:59:40,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 04:59:40,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 04:59:40,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 04:59:40,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39246.73 MB 2025-02-15 04:59:40,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47637.77 MB 2025-02-15 04:59:40,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 04:59:40,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52525.27 MB 2025-02-15 04:59:40,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62954.41 MB 2025-02-15 04:59:40,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10429.14 MB 2025-02-15 04:59:40,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47637.77 MB 2025-02-15 04:59:40,795 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 04:59:40,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:40,797 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 04:59:40,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:40,798 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 04:59:40,803 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 04:59:40,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 04:59:40,804 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 04:59:40,804 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:00:43,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:00:43,529 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:00:43,534 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:00:43,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:00:43,537 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1357, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:00:43,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:00:43,538 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1357, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:01:04,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:01:04,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:01:04,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.85 seconds 2025-02-15 05:01:04,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:04,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39552.56 MB 2025-02-15 05:01:04,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44355.04 MB 2025-02-15 05:01:04,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4802.48 MB 2025-02-15 05:01:04,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71296.88 MB 2025-02-15 05:01:04,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54458.84 MB 2025-02-15 05:01:04,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16838.03 MB 2025-02-15 05:01:04,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53327.95 MB 2025-02-15 05:01:04,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:01:04,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:01:04,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:01:04,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:04,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44355.04 MB 2025-02-15 05:01:04,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38977.31 MB 2025-02-15 05:01:04,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5377.73 MB 2025-02-15 05:01:04,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54458.84 MB 2025-02-15 05:01:04,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54458.84 MB 2025-02-15 05:01:04,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:01:04,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48007.01 MB 2025-02-15 05:01:05,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:01:05,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:01:05,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.26 seconds 2025-02-15 05:01:05,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38977.31 MB 2025-02-15 05:01:05,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39322.36 MB 2025-02-15 05:01:05,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 345.05 MB 2025-02-15 05:01:05,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54458.84 MB 2025-02-15 05:01:05,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46904.90 MB 2025-02-15 05:01:05,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7553.94 MB 2025-02-15 05:01:05,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43316.83 MB 2025-02-15 05:01:05,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:01:05,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:01:05,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:01:05,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39322.36 MB 2025-02-15 05:01:05,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40550.26 MB 2025-02-15 05:01:05,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1227.90 MB 2025-02-15 05:01:05,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46904.90 MB 2025-02-15 05:01:05,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46904.90 MB 2025-02-15 05:01:05,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:01:05,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41471.59 MB 2025-02-15 05:01:05,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:01:05,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:01:05,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 05:01:05,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40550.26 MB 2025-02-15 05:01:05,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42007.49 MB 2025-02-15 05:01:05,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1457.23 MB 2025-02-15 05:01:05,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46904.90 MB 2025-02-15 05:01:05,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47519.37 MB 2025-02-15 05:01:05,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 614.47 MB 2025-02-15 05:01:05,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45612.30 MB 2025-02-15 05:01:05,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:01:05,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:01:05,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 05:01:05,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39322.36 MB 2025-02-15 05:01:05,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42007.49 MB 2025-02-15 05:01:05,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2685.13 MB 2025-02-15 05:01:05,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46904.90 MB 2025-02-15 05:01:05,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47519.37 MB 2025-02-15 05:01:05,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 614.47 MB 2025-02-15 05:01:05,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45612.30 MB 2025-02-15 05:01:05,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:01:05,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:01:05,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:01:05,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43004.29 MB 2025-02-15 05:01:05,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43502.85 MB 2025-02-15 05:01:05,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 498.55 MB 2025-02-15 05:01:05,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47519.37 MB 2025-02-15 05:01:05,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47792.00 MB 2025-02-15 05:01:05,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 272.63 MB 2025-02-15 05:01:05,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43962.91 MB 2025-02-15 05:01:05,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:01:05,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:01:05,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:01:05,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43771.23 MB 2025-02-15 05:01:05,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43986.14 MB 2025-02-15 05:01:05,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.92 MB 2025-02-15 05:01:05,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47792.00 MB 2025-02-15 05:01:05,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47792.00 MB 2025-02-15 05:01:05,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:01:05,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44064.56 MB 2025-02-15 05:01:05,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:01:05,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:01:05,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.44 seconds 2025-02-15 05:01:05,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:05,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34824.67 MB 2025-02-15 05:01:05,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44187.22 MB 2025-02-15 05:01:05,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9362.55 MB 2025-02-15 05:01:05,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71296.88 MB 2025-02-15 05:01:05,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47792.00 MB 2025-02-15 05:01:05,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23504.88 MB 2025-02-15 05:01:05,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44187.22 MB 2025-02-15 05:01:06,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:01:06,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:01:06,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 05:01:06,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:06,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44187.22 MB 2025-02-15 05:01:06,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47201.25 MB 2025-02-15 05:01:06,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 05:01:06,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47792.00 MB 2025-02-15 05:01:06,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48999.96 MB 2025-02-15 05:01:06,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-15 05:01:06,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47502.88 MB 2025-02-15 05:01:06,283 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:01:06,284 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:01:06,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:01:06,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:01:06,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:01:06,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:01:06,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39168.36 MB 2025-02-15 05:01:06,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47607.38 MB 2025-02-15 05:01:06,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:01:06,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48999.96 MB 2025-02-15 05:01:06,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57390.66 MB 2025-02-15 05:01:06,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:01:06,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47607.38 MB 2025-02-15 05:01:06,545 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:01:06,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:06,548 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:01:06,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:06,550 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:01:06,558 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:01:06,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:06,560 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:01:06,560 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:01:33,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:33,706 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:01:33,711 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:01:33,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:33,715 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1727, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:01:33,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:01:33,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1727, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:02:00,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:02:00,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:02:00,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.84 seconds 2025-02-15 05:02:00,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:00,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42130.78 MB 2025-02-15 05:02:00,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48242.54 MB 2025-02-15 05:02:00,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.76 MB 2025-02-15 05:02:00,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69975.67 MB 2025-02-15 05:02:00,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54484.01 MB 2025-02-15 05:02:00,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15491.66 MB 2025-02-15 05:02:00,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57120.70 MB 2025-02-15 05:02:00,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:02:00,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:02:00,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 05:02:00,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:00,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48242.54 MB 2025-02-15 05:02:00,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41884.05 MB 2025-02-15 05:02:00,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6358.49 MB 2025-02-15 05:02:00,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54484.01 MB 2025-02-15 05:02:00,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67442.31 MB 2025-02-15 05:02:00,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12958.30 MB 2025-02-15 05:02:00,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65598.77 MB 2025-02-15 05:02:02,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:02:02,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:02:02,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 05:02:02,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:02,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41884.05 MB 2025-02-15 05:02:02,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42414.89 MB 2025-02-15 05:02:02,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:02:02,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67442.31 MB 2025-02-15 05:02:02,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49786.39 MB 2025-02-15 05:02:02,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17655.92 MB 2025-02-15 05:02:02,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46393.44 MB 2025-02-15 05:02:02,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:02:02,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:02:02,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:02:02,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:02,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42414.89 MB 2025-02-15 05:02:02,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44304.43 MB 2025-02-15 05:02:02,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:02:02,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49786.39 MB 2025-02-15 05:02:02,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49786.39 MB 2025-02-15 05:02:02,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:02,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45721.86 MB 2025-02-15 05:02:02,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:02:02,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:02:02,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:02:02,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:02,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44304.43 MB 2025-02-15 05:02:02,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46546.28 MB 2025-02-15 05:02:02,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:02:02,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49786.39 MB 2025-02-15 05:02:02,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54976.84 MB 2025-02-15 05:02:02,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 05:02:02,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52090.56 MB 2025-02-15 05:02:02,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:02:02,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:02:02,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 05:02:02,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:02,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42414.89 MB 2025-02-15 05:02:02,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46546.28 MB 2025-02-15 05:02:02,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:02:02,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49786.39 MB 2025-02-15 05:02:02,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54976.84 MB 2025-02-15 05:02:02,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 05:02:02,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52090.56 MB 2025-02-15 05:02:03,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:02:03,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:02:03,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:02:03,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:03,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48079.82 MB 2025-02-15 05:02:03,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48846.83 MB 2025-02-15 05:02:03,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:02:03,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54976.84 MB 2025-02-15 05:02:03,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55396.27 MB 2025-02-15 05:02:03,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:02:03,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49554.62 MB 2025-02-15 05:02:03,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:02:03,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:02:03,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:02:03,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:03,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49259.72 MB 2025-02-15 05:02:03,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49488.58 MB 2025-02-15 05:02:03,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-15 05:02:03,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55396.27 MB 2025-02-15 05:02:03,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55396.27 MB 2025-02-15 05:02:03,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:03,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49731.04 MB 2025-02-15 05:02:03,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:02:03,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:02:03,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.48 seconds 2025-02-15 05:02:03,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:03,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36113.78 MB 2025-02-15 05:02:03,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49689.36 MB 2025-02-15 05:02:03,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13575.58 MB 2025-02-15 05:02:03,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69975.67 MB 2025-02-15 05:02:03,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55396.27 MB 2025-02-15 05:02:03,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14579.40 MB 2025-02-15 05:02:03,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49731.04 MB 2025-02-15 05:02:03,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:02:03,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:02:03,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:02:03,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:03,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49689.36 MB 2025-02-15 05:02:03,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41113.60 MB 2025-02-15 05:02:03,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8575.76 MB 2025-02-15 05:02:03,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55396.27 MB 2025-02-15 05:02:03,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55396.27 MB 2025-02-15 05:02:03,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:03,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52197.34 MB 2025-02-15 05:02:03,488 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 05:02:03,488 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-15 05:02:03,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:02:03,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:02:03,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:02:03,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:03,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41113.60 MB 2025-02-15 05:02:03,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49540.10 MB 2025-02-15 05:02:03,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 05:02:03,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55396.27 MB 2025-02-15 05:02:03,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63774.39 MB 2025-02-15 05:02:03,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-15 05:02:03,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49540.10 MB 2025-02-15 05:02:03,653 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 05:02:03,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:03,655 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:02:03,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:03,656 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:02:03,660 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:02:03,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:03,661 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:02:03,661 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-15 05:02:13,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:13,840 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:02:13,845 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:02:13,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:13,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:02:13,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:13,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:02:25,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:02:25,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:02:25,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.16 seconds 2025-02-15 05:02:25,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:25,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35058.10 MB 2025-02-15 05:02:25,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37577.83 MB 2025-02-15 05:02:25,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2519.73 MB 2025-02-15 05:02:25,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76340.53 MB 2025-02-15 05:02:25,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41420.85 MB 2025-02-15 05:02:25,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34919.68 MB 2025-02-15 05:02:25,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46567.90 MB 2025-02-15 05:02:25,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:02:25,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:02:25,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:02:25,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:25,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37577.83 MB 2025-02-15 05:02:25,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36608.43 MB 2025-02-15 05:02:25,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -969.40 MB 2025-02-15 05:02:25,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41420.85 MB 2025-02-15 05:02:25,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48423.24 MB 2025-02-15 05:02:25,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7002.39 MB 2025-02-15 05:02:25,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46678.56 MB 2025-02-15 05:02:27,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:02:27,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:02:27,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:02:27,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36608.43 MB 2025-02-15 05:02:27,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37139.27 MB 2025-02-15 05:02:27,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:02:27,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48423.24 MB 2025-02-15 05:02:27,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43545.26 MB 2025-02-15 05:02:27,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4877.98 MB 2025-02-15 05:02:27,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41117.82 MB 2025-02-15 05:02:27,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:02:27,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:02:27,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:02:27,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37139.27 MB 2025-02-15 05:02:27,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39028.81 MB 2025-02-15 05:02:27,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:02:27,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43545.26 MB 2025-02-15 05:02:27,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43545.26 MB 2025-02-15 05:02:27,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:27,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40446.24 MB 2025-02-15 05:02:27,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:02:27,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:02:27,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:02:27,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39028.81 MB 2025-02-15 05:02:27,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41270.66 MB 2025-02-15 05:02:27,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:02:27,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43545.26 MB 2025-02-15 05:02:27,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49207.57 MB 2025-02-15 05:02:27,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:02:27,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46814.94 MB 2025-02-15 05:02:27,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:02:27,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:02:27,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:02:27,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37139.27 MB 2025-02-15 05:02:27,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41270.66 MB 2025-02-15 05:02:27,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:02:27,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43545.26 MB 2025-02-15 05:02:27,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49207.57 MB 2025-02-15 05:02:27,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:02:27,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46814.94 MB 2025-02-15 05:02:27,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:02:27,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:02:27,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:02:27,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42804.20 MB 2025-02-15 05:02:27,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43571.21 MB 2025-02-15 05:02:27,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:02:27,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49207.57 MB 2025-02-15 05:02:27,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49627.00 MB 2025-02-15 05:02:27,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:02:27,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44278.99 MB 2025-02-15 05:02:27,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:02:27,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:02:27,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:02:27,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43984.10 MB 2025-02-15 05:02:27,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44213.13 MB 2025-02-15 05:02:27,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-15 05:02:27,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49627.00 MB 2025-02-15 05:02:27,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49627.00 MB 2025-02-15 05:02:27,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:27,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44442.51 MB 2025-02-15 05:02:27,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:02:27,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:02:27,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.56 seconds 2025-02-15 05:02:27,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32577.44 MB 2025-02-15 05:02:27,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44414.20 MB 2025-02-15 05:02:27,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11836.77 MB 2025-02-15 05:02:27,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76340.53 MB 2025-02-15 05:02:27,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49627.00 MB 2025-02-15 05:02:27,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26713.52 MB 2025-02-15 05:02:27,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44442.51 MB 2025-02-15 05:02:27,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:02:27,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:02:27,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:02:27,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44414.20 MB 2025-02-15 05:02:27,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37581.83 MB 2025-02-15 05:02:27,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6832.38 MB 2025-02-15 05:02:27,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49627.00 MB 2025-02-15 05:02:27,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49627.00 MB 2025-02-15 05:02:27,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:27,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46925.87 MB 2025-02-15 05:02:27,701 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:02:27,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:02:27,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:02:27,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:02:27,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:02:27,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:27,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37581.83 MB 2025-02-15 05:02:27,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46020.85 MB 2025-02-15 05:02:27,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:02:27,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49627.00 MB 2025-02-15 05:02:27,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58017.71 MB 2025-02-15 05:02:27,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:02:27,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46020.85 MB 2025-02-15 05:02:27,869 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:02:27,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:27,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:02:27,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:27,872 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:02:27,877 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:02:27,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:27,878 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:02:27,878 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:02:57,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:57,083 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:02:57,088 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:02:57,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:57,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:02:57,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:02:57,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:02:59,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:02:59,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:02:59,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.81 seconds 2025-02-15 05:02:59,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:59,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31358.01 MB 2025-02-15 05:02:59,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31998.56 MB 2025-02-15 05:02:59,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-15 05:02:59,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70602.72 MB 2025-02-15 05:02:59,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 05:02:59,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33139.20 MB 2025-02-15 05:02:59,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40829.38 MB 2025-02-15 05:02:59,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:02:59,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:02:59,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:02:59,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:02:59,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31998.56 MB 2025-02-15 05:02:59,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32280.81 MB 2025-02-15 05:02:59,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 282.25 MB 2025-02-15 05:02:59,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 05:02:59,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 05:02:59,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:02:59,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34484.77 MB 2025-02-15 05:03:00,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:03:00,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:03:00,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 05:03:00,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32280.81 MB 2025-02-15 05:03:00,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32515.71 MB 2025-02-15 05:03:00,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 05:03:00,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 05:03:00,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 05:03:00,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:00,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36451.53 MB 2025-02-15 05:03:00,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:03:00,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:03:00,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:03:00,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32515.64 MB 2025-02-15 05:03:00,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33351.56 MB 2025-02-15 05:03:00,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 05:03:00,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 05:03:00,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 05:03:00,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:00,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33978.78 MB 2025-02-15 05:03:00,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:03:00,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:03:00,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:03:00,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33351.56 MB 2025-02-15 05:03:00,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34343.62 MB 2025-02-15 05:03:00,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 05:03:00,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 05:03:00,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38721.81 MB 2025-02-15 05:03:00,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-15 05:03:00,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36796.92 MB 2025-02-15 05:03:00,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:03:00,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:03:00,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:03:00,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32515.64 MB 2025-02-15 05:03:00,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34343.62 MB 2025-02-15 05:03:00,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 05:03:00,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 05:03:00,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38721.81 MB 2025-02-15 05:03:00,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-15 05:03:00,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36796.92 MB 2025-02-15 05:03:00,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:03:00,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:03:00,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:03:00,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35022.21 MB 2025-02-15 05:03:00,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35361.61 MB 2025-02-15 05:03:00,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-15 05:03:00,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38721.81 MB 2025-02-15 05:03:00,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38906.36 MB 2025-02-15 05:03:00,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-15 05:03:00,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35681.51 MB 2025-02-15 05:03:00,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:03:00,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:03:00,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:03:00,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35544.32 MB 2025-02-15 05:03:00,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35773.00 MB 2025-02-15 05:03:00,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-15 05:03:00,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38906.36 MB 2025-02-15 05:03:00,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38906.36 MB 2025-02-15 05:03:00,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:00,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35812.26 MB 2025-02-15 05:03:00,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:03:00,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:03:00,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.87 seconds 2025-02-15 05:03:00,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:00,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30727.39 MB 2025-02-15 05:03:00,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35974.08 MB 2025-02-15 05:03:00,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5246.69 MB 2025-02-15 05:03:00,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70602.72 MB 2025-02-15 05:03:00,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38906.36 MB 2025-02-15 05:03:00,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31696.36 MB 2025-02-15 05:03:00,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35974.08 MB 2025-02-15 05:03:01,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:03:01,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:03:01,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:03:01,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:01,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35974.08 MB 2025-02-15 05:03:01,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34679.38 MB 2025-02-15 05:03:01,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1294.70 MB 2025-02-15 05:03:01,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38906.36 MB 2025-02-15 05:03:01,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38906.36 MB 2025-02-15 05:03:01,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:01,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36209.98 MB 2025-02-15 05:03:01,256 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:03:01,256 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:03:01,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:03:01,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:03:01,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:03:01,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:01,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34679.38 MB 2025-02-15 05:03:01,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43119.28 MB 2025-02-15 05:03:01,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.91 MB 2025-02-15 05:03:01,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38906.36 MB 2025-02-15 05:03:01,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47297.07 MB 2025-02-15 05:03:01,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:03:01,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43119.28 MB 2025-02-15 05:03:01,427 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:03:01,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:01,428 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:03:01,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:01,429 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:03:01,434 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:03:01,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:01,435 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:03:01,435 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:03:43,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:43,960 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:03:43,967 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:03:43,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:43,974 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:03:43,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:43,976 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:03:54,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:03:54,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:03:54,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.48 seconds 2025-02-15 05:03:54,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:54,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34814.22 MB 2025-02-15 05:03:54,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37210.08 MB 2025-02-15 05:03:54,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2395.87 MB 2025-02-15 05:03:54,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59882.08 MB 2025-02-15 05:03:54,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42402.32 MB 2025-02-15 05:03:54,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17479.76 MB 2025-02-15 05:03:54,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46097.53 MB 2025-02-15 05:03:54,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:03:54,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:03:54,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 05:03:54,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:54,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37210.08 MB 2025-02-15 05:03:54,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36426.48 MB 2025-02-15 05:03:54,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -783.60 MB 2025-02-15 05:03:54,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42402.32 MB 2025-02-15 05:03:54,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48836.38 MB 2025-02-15 05:03:54,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6434.06 MB 2025-02-15 05:03:54,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45741.03 MB 2025-02-15 05:03:56,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:03:56,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:03:56,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:03:56,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36426.48 MB 2025-02-15 05:03:56,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36957.32 MB 2025-02-15 05:03:56,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:03:56,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48836.38 MB 2025-02-15 05:03:56,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42129.69 MB 2025-02-15 05:03:56,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6706.69 MB 2025-02-15 05:03:56,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40935.87 MB 2025-02-15 05:03:56,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:03:56,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:03:56,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:03:56,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36957.32 MB 2025-02-15 05:03:56,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38846.85 MB 2025-02-15 05:03:56,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:03:56,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42129.69 MB 2025-02-15 05:03:56,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43073.40 MB 2025-02-15 05:03:56,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 05:03:56,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40264.28 MB 2025-02-15 05:03:56,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:03:56,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:03:56,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 05:03:56,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38846.85 MB 2025-02-15 05:03:56,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41088.71 MB 2025-02-15 05:03:56,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:03:56,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43073.40 MB 2025-02-15 05:03:56,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48735.72 MB 2025-02-15 05:03:56,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:03:56,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46632.99 MB 2025-02-15 05:03:56,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:03:56,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:03:56,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 05:03:56,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36957.32 MB 2025-02-15 05:03:56,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41088.71 MB 2025-02-15 05:03:56,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:03:56,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42129.69 MB 2025-02-15 05:03:56,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48735.72 MB 2025-02-15 05:03:56,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 05:03:56,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46632.99 MB 2025-02-15 05:03:56,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:03:56,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:03:56,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 05:03:56,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42622.25 MB 2025-02-15 05:03:56,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43389.25 MB 2025-02-15 05:03:56,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:03:56,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48735.72 MB 2025-02-15 05:03:56,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49155.15 MB 2025-02-15 05:03:56,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:03:56,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44097.04 MB 2025-02-15 05:03:56,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:03:56,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:03:56,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:03:56,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43802.14 MB 2025-02-15 05:03:56,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44034.96 MB 2025-02-15 05:03:56,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.82 MB 2025-02-15 05:03:56,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49155.15 MB 2025-02-15 05:03:56,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49155.15 MB 2025-02-15 05:03:56,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:56,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44213.14 MB 2025-02-15 05:03:56,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:03:56,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:03:56,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.97 seconds 2025-02-15 05:03:56,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:56,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32455.49 MB 2025-02-15 05:03:56,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44236.03 MB 2025-02-15 05:03:56,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11780.54 MB 2025-02-15 05:03:56,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59882.08 MB 2025-02-15 05:03:56,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49155.15 MB 2025-02-15 05:03:56,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10726.93 MB 2025-02-15 05:03:56,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44236.03 MB 2025-02-15 05:03:57,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:03:57,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:03:57,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:03:57,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:57,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44236.03 MB 2025-02-15 05:03:57,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37460.52 MB 2025-02-15 05:03:57,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6775.51 MB 2025-02-15 05:03:57,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49155.15 MB 2025-02-15 05:03:57,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49155.15 MB 2025-02-15 05:03:57,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:03:57,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46748.34 MB 2025-02-15 05:03:57,236 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:03:57,236 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:03:57,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:03:57,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:03:57,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:03:57,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:03:57,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37460.52 MB 2025-02-15 05:03:57,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45899.55 MB 2025-02-15 05:03:57,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:03:57,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49155.15 MB 2025-02-15 05:03:57,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57545.85 MB 2025-02-15 05:03:57,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:03:57,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45899.55 MB 2025-02-15 05:03:57,409 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:03:57,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:57,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:03:57,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:57,412 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:03:57,417 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:03:57,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:03:57,418 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:03:57,418 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:04:45,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:04:45,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:04:45,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:04:45,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:04:45,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 963, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:04:45,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:04:45,122 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 963, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:05:00,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:05:00,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:05:00,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.89 seconds 2025-02-15 05:05:00,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:00,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36807.11 MB 2025-02-15 05:05:00,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40215.11 MB 2025-02-15 05:05:00,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3408.00 MB 2025-02-15 05:05:00,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70130.86 MB 2025-02-15 05:05:00,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47609.54 MB 2025-02-15 05:05:00,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22521.32 MB 2025-02-15 05:05:00,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49222.88 MB 2025-02-15 05:05:00,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:05:00,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:05:00,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:05:00,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:00,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40215.11 MB 2025-02-15 05:05:00,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37912.25 MB 2025-02-15 05:05:00,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2302.86 MB 2025-02-15 05:05:00,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47609.54 MB 2025-02-15 05:05:00,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54360.28 MB 2025-02-15 05:05:00,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6750.73 MB 2025-02-15 05:05:00,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51190.63 MB 2025-02-15 05:05:02,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:05:02,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:05:02,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:05:02,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37912.25 MB 2025-02-15 05:05:02,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38443.09 MB 2025-02-15 05:05:02,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:05:02,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54360.28 MB 2025-02-15 05:05:02,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44199.58 MB 2025-02-15 05:05:02,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10160.70 MB 2025-02-15 05:05:02,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42421.64 MB 2025-02-15 05:05:02,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:05:02,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:05:02,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:05:02,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38443.09 MB 2025-02-15 05:05:02,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40332.63 MB 2025-02-15 05:05:02,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:05:02,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44199.58 MB 2025-02-15 05:05:02,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44199.58 MB 2025-02-15 05:05:02,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:05:02,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41750.06 MB 2025-02-15 05:05:02,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:05:02,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:05:02,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:05:02,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40332.63 MB 2025-02-15 05:05:02,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42574.48 MB 2025-02-15 05:05:02,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:05:02,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44199.58 MB 2025-02-15 05:05:02,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50333.75 MB 2025-02-15 05:05:02,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:05:02,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48118.77 MB 2025-02-15 05:05:02,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:05:02,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:05:02,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:05:02,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38443.09 MB 2025-02-15 05:05:02,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42574.48 MB 2025-02-15 05:05:02,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:05:02,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44199.58 MB 2025-02-15 05:05:02,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50333.75 MB 2025-02-15 05:05:02,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:05:02,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48118.77 MB 2025-02-15 05:05:02,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:05:02,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:05:02,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:05:02,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44108.03 MB 2025-02-15 05:05:02,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44875.03 MB 2025-02-15 05:05:02,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:05:02,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50333.75 MB 2025-02-15 05:05:02,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50753.18 MB 2025-02-15 05:05:02,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:05:02,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45582.82 MB 2025-02-15 05:05:02,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:05:02,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:05:02,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:05:02,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45287.92 MB 2025-02-15 05:05:02,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45516.56 MB 2025-02-15 05:05:02,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-15 05:05:02,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50753.18 MB 2025-02-15 05:05:02,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50753.18 MB 2025-02-15 05:05:02,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:05:02,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45754.02 MB 2025-02-15 05:05:02,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:05:02,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:05:02,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.29 seconds 2025-02-15 05:05:02,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33451.94 MB 2025-02-15 05:05:02,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45717.12 MB 2025-02-15 05:05:02,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12265.17 MB 2025-02-15 05:05:02,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70130.86 MB 2025-02-15 05:05:02,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50753.18 MB 2025-02-15 05:05:02,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19377.68 MB 2025-02-15 05:05:02,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45754.02 MB 2025-02-15 05:05:02,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:05:02,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:05:02,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:05:02,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45717.12 MB 2025-02-15 05:05:02,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38448.33 MB 2025-02-15 05:05:02,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7268.78 MB 2025-02-15 05:05:02,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50753.18 MB 2025-02-15 05:05:02,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50753.18 MB 2025-02-15 05:05:02,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:05:02,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48222.33 MB 2025-02-15 05:05:02,708 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 05:05:02,709 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:05:02,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:05:02,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:05:02,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:05:02,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:05:02,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38448.33 MB 2025-02-15 05:05:02,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46866.07 MB 2025-02-15 05:05:02,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-15 05:05:02,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50753.18 MB 2025-02-15 05:05:02,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59120.81 MB 2025-02-15 05:05:02,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 05:05:02,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46866.07 MB 2025-02-15 05:05:02,878 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 05:05:02,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:02,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:05:02,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:02,881 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:05:02,886 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:05:02,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:02,887 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:05:02,887 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:05:50,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:50,381 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:05:50,386 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:05:50,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:50,390 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:05:50,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:05:50,391 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:06:10,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:06:10,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:06:10,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.21 seconds 2025-02-15 05:06:10,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:10,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39162.35 MB 2025-02-15 05:06:10,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43766.51 MB 2025-02-15 05:06:10,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4604.17 MB 2025-02-15 05:06:10,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67488.45 MB 2025-02-15 05:06:10,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52967.77 MB 2025-02-15 05:06:10,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14520.68 MB 2025-02-15 05:06:10,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52710.84 MB 2025-02-15 05:06:10,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:06:10,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:06:10,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:06:10,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:10,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43766.51 MB 2025-02-15 05:06:10,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39669.41 MB 2025-02-15 05:06:10,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4097.10 MB 2025-02-15 05:06:10,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52967.77 MB 2025-02-15 05:06:10,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62023.27 MB 2025-02-15 05:06:10,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9055.50 MB 2025-02-15 05:06:10,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57382.18 MB 2025-02-15 05:06:12,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:06:12,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:06:12,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:06:12,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:12,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39669.41 MB 2025-02-15 05:06:12,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40200.25 MB 2025-02-15 05:06:12,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:06:12,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62023.27 MB 2025-02-15 05:06:12,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44178.60 MB 2025-02-15 05:06:12,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17844.67 MB 2025-02-15 05:06:12,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44179.84 MB 2025-02-15 05:06:12,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:06:12,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:06:12,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:06:12,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:12,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40200.25 MB 2025-02-15 05:06:12,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42089.78 MB 2025-02-15 05:06:12,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:06:12,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44178.60 MB 2025-02-15 05:06:12,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46066.04 MB 2025-02-15 05:06:12,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 05:06:12,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43507.21 MB 2025-02-15 05:06:12,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:06:12,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:06:12,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:06:12,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:12,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42089.78 MB 2025-02-15 05:06:12,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44331.64 MB 2025-02-15 05:06:12,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:06:12,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46066.04 MB 2025-02-15 05:06:12,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51728.35 MB 2025-02-15 05:06:12,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:06:12,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49875.92 MB 2025-02-15 05:06:12,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:06:12,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:06:12,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:06:12,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:12,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40200.25 MB 2025-02-15 05:06:12,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44331.64 MB 2025-02-15 05:06:12,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:06:12,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44178.60 MB 2025-02-15 05:06:12,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51728.35 MB 2025-02-15 05:06:12,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 05:06:12,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49875.92 MB 2025-02-15 05:06:13,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:06:13,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:06:13,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:06:13,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:13,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45865.18 MB 2025-02-15 05:06:13,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46632.18 MB 2025-02-15 05:06:13,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:06:13,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51728.35 MB 2025-02-15 05:06:13,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52147.78 MB 2025-02-15 05:06:13,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:06:13,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47339.97 MB 2025-02-15 05:06:13,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:06:13,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:06:13,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:06:13,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:13,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47045.07 MB 2025-02-15 05:06:13,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47272.86 MB 2025-02-15 05:06:13,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.78 MB 2025-02-15 05:06:13,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52147.78 MB 2025-02-15 05:06:13,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52147.78 MB 2025-02-15 05:06:13,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:06:13,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47507.08 MB 2025-02-15 05:06:13,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:06:13,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:06:13,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.65 seconds 2025-02-15 05:06:13,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:13,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34629.56 MB 2025-02-15 05:06:13,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47473.83 MB 2025-02-15 05:06:13,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12844.27 MB 2025-02-15 05:06:13,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67488.45 MB 2025-02-15 05:06:13,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52147.78 MB 2025-02-15 05:06:13,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15340.67 MB 2025-02-15 05:06:13,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47507.08 MB 2025-02-15 05:06:13,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:06:13,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:06:13,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:06:13,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:13,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47473.83 MB 2025-02-15 05:06:13,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39632.43 MB 2025-02-15 05:06:13,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7841.40 MB 2025-02-15 05:06:13,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52147.78 MB 2025-02-15 05:06:13,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52147.78 MB 2025-02-15 05:06:13,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:06:13,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49984.27 MB 2025-02-15 05:06:13,326 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 05:06:13,327 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:06:13,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:06:13,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:06:13,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:06:13,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:06:13,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39632.43 MB 2025-02-15 05:06:13,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48067.27 MB 2025-02-15 05:06:13,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 05:06:13,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52147.78 MB 2025-02-15 05:06:13,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60534.29 MB 2025-02-15 05:06:13,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 05:06:13,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48067.27 MB 2025-02-15 05:06:13,495 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 05:06:13,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:06:13,497 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:06:13,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:06:13,498 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:06:13,503 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:06:13,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:06:13,504 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:06:13,504 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:07:40,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:07:40,264 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:07:40,269 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:07:40,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:07:40,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1449, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:07:40,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:07:40,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1449, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:08:02,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:08:02,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:08:02,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.34 seconds 2025-02-15 05:08:02,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:02,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40193.64 MB 2025-02-15 05:08:02,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45321.57 MB 2025-02-15 05:08:02,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5127.93 MB 2025-02-15 05:08:02,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73113.01 MB 2025-02-15 05:08:02,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53500.44 MB 2025-02-15 05:08:02,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-15 05:08:02,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54195.12 MB 2025-02-15 05:08:02,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:08:02,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:08:02,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:08:02,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:02,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45321.57 MB 2025-02-15 05:08:02,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40438.82 MB 2025-02-15 05:08:02,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4882.75 MB 2025-02-15 05:08:02,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53500.44 MB 2025-02-15 05:08:02,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61075.36 MB 2025-02-15 05:08:02,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7574.91 MB 2025-02-15 05:08:02,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57118.09 MB 2025-02-15 05:08:04,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:08:04,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:08:04,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:08:04,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:04,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40438.82 MB 2025-02-15 05:08:04,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40969.66 MB 2025-02-15 05:08:04,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:08:04,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61075.36 MB 2025-02-15 05:08:04,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48370.81 MB 2025-02-15 05:08:04,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12704.55 MB 2025-02-15 05:08:04,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44948.20 MB 2025-02-15 05:08:04,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:08:04,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:08:04,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:08:04,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:04,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40969.66 MB 2025-02-15 05:08:04,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42859.19 MB 2025-02-15 05:08:04,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:08:04,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48370.81 MB 2025-02-15 05:08:04,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48370.81 MB 2025-02-15 05:08:04,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:08:04,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44276.62 MB 2025-02-15 05:08:04,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:08:04,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:08:04,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:08:04,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:04,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42859.19 MB 2025-02-15 05:08:04,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45101.05 MB 2025-02-15 05:08:04,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:08:04,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48370.81 MB 2025-02-15 05:08:04,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53089.40 MB 2025-02-15 05:08:04,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:08:04,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50645.33 MB 2025-02-15 05:08:04,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:08:04,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:08:04,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:08:04,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:04,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40969.66 MB 2025-02-15 05:08:04,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45101.05 MB 2025-02-15 05:08:04,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:08:04,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48370.81 MB 2025-02-15 05:08:04,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53089.40 MB 2025-02-15 05:08:04,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:08:04,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50645.33 MB 2025-02-15 05:08:05,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:08:05,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:08:05,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:08:05,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:05,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46634.59 MB 2025-02-15 05:08:05,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47401.59 MB 2025-02-15 05:08:05,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:08:05,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53089.40 MB 2025-02-15 05:08:05,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53508.83 MB 2025-02-15 05:08:05,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:08:05,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48109.38 MB 2025-02-15 05:08:05,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:08:05,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:08:05,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:08:05,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:05,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47814.48 MB 2025-02-15 05:08:05,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48041.99 MB 2025-02-15 05:08:05,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.51 MB 2025-02-15 05:08:05,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53508.83 MB 2025-02-15 05:08:05,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53508.83 MB 2025-02-15 05:08:05,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:08:05,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48268.80 MB 2025-02-15 05:08:05,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:08:05,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:08:05,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.77 seconds 2025-02-15 05:08:05,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:05,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35145.20 MB 2025-02-15 05:08:05,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48242.03 MB 2025-02-15 05:08:05,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13096.83 MB 2025-02-15 05:08:05,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73113.01 MB 2025-02-15 05:08:05,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53508.83 MB 2025-02-15 05:08:05,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19604.18 MB 2025-02-15 05:08:05,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48268.80 MB 2025-02-15 05:08:05,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:08:05,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:08:05,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:08:05,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:05,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48242.03 MB 2025-02-15 05:08:05,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40133.59 MB 2025-02-15 05:08:05,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8108.44 MB 2025-02-15 05:08:05,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53508.83 MB 2025-02-15 05:08:05,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53508.83 MB 2025-02-15 05:08:05,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:08:05,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50740.80 MB 2025-02-15 05:08:05,337 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 05:08:05,337 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:08:05,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:08:05,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:08:05,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:08:05,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:08:05,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40133.59 MB 2025-02-15 05:08:05,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48529.84 MB 2025-02-15 05:08:05,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.25 MB 2025-02-15 05:08:05,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53508.83 MB 2025-02-15 05:08:05,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57682.17 MB 2025-02-15 05:08:05,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 05:08:05,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48529.84 MB 2025-02-15 05:08:05,503 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 05:08:05,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:08:05,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:08:05,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:08:05,505 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:08:05,510 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:08:05,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:08:05,511 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:08:05,511 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:09:09,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:09,473 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:09:09,478 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:09:09,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:09,482 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1853, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:09:09,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:09,483 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1853, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:09:38,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:09:38,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:09:38,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.59 seconds 2025-02-15 05:09:38,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:38,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43008.77 MB 2025-02-15 05:09:38,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49566.57 MB 2025-02-15 05:09:38,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6557.79 MB 2025-02-15 05:09:38,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66028.83 MB 2025-02-15 05:09:38,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59072.58 MB 2025-02-15 05:09:38,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6956.25 MB 2025-02-15 05:09:38,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58368.95 MB 2025-02-15 05:09:38,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:09:38,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:09:38,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 05:09:38,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:38,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49566.57 MB 2025-02-15 05:09:38,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42539.09 MB 2025-02-15 05:09:38,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7027.48 MB 2025-02-15 05:09:38,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59072.58 MB 2025-02-15 05:09:38,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71689.04 MB 2025-02-15 05:09:38,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12616.47 MB 2025-02-15 05:09:38,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68537.84 MB 2025-02-15 05:09:40,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:09:40,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:09:40,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:09:40,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42539.09 MB 2025-02-15 05:09:40,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43069.93 MB 2025-02-15 05:09:40,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:09:40,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71689.04 MB 2025-02-15 05:09:40,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52514.78 MB 2025-02-15 05:09:40,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19174.26 MB 2025-02-15 05:09:40,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47048.47 MB 2025-02-15 05:09:40,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:09:40,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:09:40,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:09:40,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43069.93 MB 2025-02-15 05:09:40,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44959.46 MB 2025-02-15 05:09:40,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:09:40,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52514.78 MB 2025-02-15 05:09:40,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52514.78 MB 2025-02-15 05:09:40,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:09:40,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46376.89 MB 2025-02-15 05:09:40,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:09:40,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:09:40,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:09:40,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44959.46 MB 2025-02-15 05:09:40,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47201.32 MB 2025-02-15 05:09:40,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:09:40,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52514.78 MB 2025-02-15 05:09:40,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55345.94 MB 2025-02-15 05:09:40,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:09:40,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52745.60 MB 2025-02-15 05:09:40,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:09:40,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:09:40,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:09:40,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43069.93 MB 2025-02-15 05:09:40,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47201.32 MB 2025-02-15 05:09:40,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:09:40,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52514.78 MB 2025-02-15 05:09:40,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55345.94 MB 2025-02-15 05:09:40,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:09:40,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52745.60 MB 2025-02-15 05:09:40,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:09:40,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:09:40,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:09:40,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48734.86 MB 2025-02-15 05:09:40,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49501.86 MB 2025-02-15 05:09:40,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:09:40,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55345.94 MB 2025-02-15 05:09:40,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55765.37 MB 2025-02-15 05:09:40,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:09:40,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50209.65 MB 2025-02-15 05:09:40,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:09:40,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:09:40,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:09:40,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49914.75 MB 2025-02-15 05:09:40,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50143.91 MB 2025-02-15 05:09:40,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-15 05:09:40,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55765.37 MB 2025-02-15 05:09:40,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55765.37 MB 2025-02-15 05:09:40,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:09:40,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50358.99 MB 2025-02-15 05:09:40,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:09:40,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:09:40,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.06 seconds 2025-02-15 05:09:40,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36552.77 MB 2025-02-15 05:09:40,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50344.98 MB 2025-02-15 05:09:40,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13792.21 MB 2025-02-15 05:09:40,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66028.83 MB 2025-02-15 05:09:40,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55765.37 MB 2025-02-15 05:09:40,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10263.46 MB 2025-02-15 05:09:40,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50358.99 MB 2025-02-15 05:09:40,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:09:40,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:09:40,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:09:40,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50344.98 MB 2025-02-15 05:09:40,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41557.16 MB 2025-02-15 05:09:40,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8787.82 MB 2025-02-15 05:09:40,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55765.37 MB 2025-02-15 05:09:40,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55765.37 MB 2025-02-15 05:09:40,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:09:40,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52856.65 MB 2025-02-15 05:09:40,836 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:09:40,837 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:09:40,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:09:40,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:09:40,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:09:40,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:09:40,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41557.16 MB 2025-02-15 05:09:40,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49996.18 MB 2025-02-15 05:09:40,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:09:40,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55765.37 MB 2025-02-15 05:09:40,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64156.07 MB 2025-02-15 05:09:40,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:09:40,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49996.18 MB 2025-02-15 05:09:41,001 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:09:41,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:41,002 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:09:41,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:41,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:09:41,008 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:09:41,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:41,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:09:41,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:09:50,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:50,584 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:09:50,588 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:09:50,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:50,592 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1587, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:09:50,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:09:50,593 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1587, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:10:15,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:10:15,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:10:15,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.88 seconds 2025-02-15 05:10:15,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:15,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41155.24 MB 2025-02-15 05:10:15,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46771.55 MB 2025-02-15 05:10:15,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5616.30 MB 2025-02-15 05:10:15,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76741.08 MB 2025-02-15 05:10:15,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53959.72 MB 2025-02-15 05:10:15,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22781.36 MB 2025-02-15 05:10:15,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55610.25 MB 2025-02-15 05:10:15,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:10:15,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:10:15,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:10:15,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:15,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46771.55 MB 2025-02-15 05:10:15,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39126.58 MB 2025-02-15 05:10:15,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7644.97 MB 2025-02-15 05:10:15,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53959.72 MB 2025-02-15 05:10:15,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53959.72 MB 2025-02-15 05:10:15,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:15,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48330.66 MB 2025-02-15 05:10:16,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:10:16,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:10:16,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-15 05:10:16,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39126.58 MB 2025-02-15 05:10:16,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39273.89 MB 2025-02-15 05:10:16,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.31 MB 2025-02-15 05:10:16,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53959.72 MB 2025-02-15 05:10:16,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:10:16,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5618.27 MB 2025-02-15 05:10:16,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43211.30 MB 2025-02-15 05:10:16,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:10:16,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:10:16,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:10:16,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39273.89 MB 2025-02-15 05:10:16,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39798.11 MB 2025-02-15 05:10:16,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 524.22 MB 2025-02-15 05:10:16,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:10:16,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:10:16,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:16,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40191.45 MB 2025-02-15 05:10:16,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:10:16,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:10:16,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:10:16,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39798.11 MB 2025-02-15 05:10:16,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40435.80 MB 2025-02-15 05:10:16,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.69 MB 2025-02-15 05:10:16,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:10:16,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:10:16,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:16,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41959.74 MB 2025-02-15 05:10:16,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:10:16,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:10:16,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:10:16,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39273.89 MB 2025-02-15 05:10:16,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40435.80 MB 2025-02-15 05:10:16,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1161.91 MB 2025-02-15 05:10:16,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:10:16,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:10:16,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:16,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41959.74 MB 2025-02-15 05:10:16,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:10:16,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:10:16,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:10:16,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41050.50 MB 2025-02-15 05:10:16,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41317.90 MB 2025-02-15 05:10:16,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.40 MB 2025-02-15 05:10:16,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:10:16,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48513.42 MB 2025-02-15 05:10:16,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-15 05:10:16,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41514.31 MB 2025-02-15 05:10:16,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:10:16,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:10:16,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:10:16,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41487.04 MB 2025-02-15 05:10:16,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41715.66 MB 2025-02-15 05:10:16,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-15 05:10:16,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 05:10:16,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48513.42 MB 2025-02-15 05:10:16,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:16,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41715.66 MB 2025-02-15 05:10:16,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:10:16,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:10:16,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.66 seconds 2025-02-15 05:10:16,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35626.01 MB 2025-02-15 05:10:16,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41916.17 MB 2025-02-15 05:10:16,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6290.16 MB 2025-02-15 05:10:16,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76741.08 MB 2025-02-15 05:10:16,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48513.42 MB 2025-02-15 05:10:16,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28227.67 MB 2025-02-15 05:10:16,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41916.17 MB 2025-02-15 05:10:16,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:10:16,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:10:16,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:10:16,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41916.17 MB 2025-02-15 05:10:16,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44921.72 MB 2025-02-15 05:10:16,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-15 05:10:16,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 05:10:16,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48513.42 MB 2025-02-15 05:10:16,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:16,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45222.24 MB 2025-02-15 05:10:16,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 05:10:16,544 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:10:16,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:10:16,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:10:16,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:10:16,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:16,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39359.19 MB 2025-02-15 05:10:16,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47774.14 MB 2025-02-15 05:10:16,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 05:10:16,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 05:10:16,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56881.05 MB 2025-02-15 05:10:16,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 05:10:16,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47774.14 MB 2025-02-15 05:10:16,708 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 05:10:16,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:16,709 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:10:16,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:16,710 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:10:16,715 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:10:16,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:16,716 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:10:16,716 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:10:27,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:27,127 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:10:27,132 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:10:27,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:27,136 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:10:27,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:27,137 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:10:29,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:10:29,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:10:29,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.53 seconds 2025-02-15 05:10:29,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:29,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31225.61 MB 2025-02-15 05:10:29,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31798.92 MB 2025-02-15 05:10:29,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 573.31 MB 2025-02-15 05:10:29,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65248.69 MB 2025-02-15 05:10:29,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:29,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25264.39 MB 2025-02-15 05:10:29,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40696.98 MB 2025-02-15 05:10:29,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:10:29,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:10:29,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:10:29,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:29,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31798.92 MB 2025-02-15 05:10:29,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31964.32 MB 2025-02-15 05:10:29,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 165.40 MB 2025-02-15 05:10:29,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:29,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:29,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:29,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33849.71 MB 2025-02-15 05:10:30,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:10:30,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:10:30,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-15 05:10:30,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31964.32 MB 2025-02-15 05:10:30,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32158.08 MB 2025-02-15 05:10:30,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-15 05:10:30,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:30,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:30,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:30,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36133.97 MB 2025-02-15 05:10:30,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:10:30,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:10:30,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:10:30,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32158.01 MB 2025-02-15 05:10:30,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32847.53 MB 2025-02-15 05:10:30,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-15 05:10:30,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:30,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:30,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:30,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33364.89 MB 2025-02-15 05:10:30,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:10:30,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:10:30,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:10:30,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32847.53 MB 2025-02-15 05:10:30,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33665.84 MB 2025-02-15 05:10:30,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-15 05:10:30,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:30,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:30,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:30,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35689.47 MB 2025-02-15 05:10:30,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:10:30,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:10:30,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:10:30,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32158.01 MB 2025-02-15 05:10:30,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33665.84 MB 2025-02-15 05:10:30,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-15 05:10:30,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:30,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-15 05:10:30,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:30,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35689.47 MB 2025-02-15 05:10:30,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:10:30,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:10:30,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:10:30,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34225.59 MB 2025-02-15 05:10:30,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34505.54 MB 2025-02-15 05:10:30,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-15 05:10:30,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-15 05:10:30,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40137.39 MB 2025-02-15 05:10:30,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 05:10:30,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34775.40 MB 2025-02-15 05:10:30,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:10:30,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:10:30,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:10:30,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34656.26 MB 2025-02-15 05:10:30,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34860.38 MB 2025-02-15 05:10:30,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.12 MB 2025-02-15 05:10:30,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40137.39 MB 2025-02-15 05:10:30,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40141.59 MB 2025-02-15 05:10:30,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 05:10:30,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34875.28 MB 2025-02-15 05:10:30,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:10:30,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:10:30,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.42 seconds 2025-02-15 05:10:30,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30661.19 MB 2025-02-15 05:10:30,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35061.13 MB 2025-02-15 05:10:30,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4399.94 MB 2025-02-15 05:10:30,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65248.69 MB 2025-02-15 05:10:30,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40141.59 MB 2025-02-15 05:10:30,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25107.10 MB 2025-02-15 05:10:30,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35061.13 MB 2025-02-15 05:10:30,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:10:30,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:10:30,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:10:30,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35061.13 MB 2025-02-15 05:10:30,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34461.84 MB 2025-02-15 05:10:30,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -599.29 MB 2025-02-15 05:10:30,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40141.59 MB 2025-02-15 05:10:30,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40141.59 MB 2025-02-15 05:10:30,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:10:30,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36164.51 MB 2025-02-15 05:10:30,847 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 05:10:30,848 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:10:30,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:10:30,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:10:30,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:10:30,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:10:30,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34461.84 MB 2025-02-15 05:10:30,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42888.02 MB 2025-02-15 05:10:30,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 05:10:30,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40141.59 MB 2025-02-15 05:10:30,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48517.61 MB 2025-02-15 05:10:30,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 05:10:30,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42888.02 MB 2025-02-15 05:10:31,014 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 05:10:31,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:31,016 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:10:31,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:31,017 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:10:31,021 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:10:31,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:10:31,022 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:10:31,023 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:11:23,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:23,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:11:23,116 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:11:23,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:23,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:11:23,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:23,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:11:25,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:11:25,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:11:25,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-15 05:11:25,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:25,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31378.91 MB 2025-02-15 05:11:25,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32030.08 MB 2025-02-15 05:11:25,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-15 05:11:25,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56893.64 MB 2025-02-15 05:11:25,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-15 05:11:25,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21082.67 MB 2025-02-15 05:11:25,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40850.28 MB 2025-02-15 05:11:25,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:11:25,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:11:25,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:11:25,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:25,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32030.08 MB 2025-02-15 05:11:25,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32226.18 MB 2025-02-15 05:11:25,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.10 MB 2025-02-15 05:11:25,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35810.97 MB 2025-02-15 05:11:25,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-15 05:11:25,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:11:25,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.83 MB 2025-02-15 05:11:26,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:11:26,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:11:26,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 05:11:26,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:26,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32226.18 MB 2025-02-15 05:11:26,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32447.80 MB 2025-02-15 05:11:26,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-15 05:11:26,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35810.97 MB 2025-02-15 05:11:26,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36213.62 MB 2025-02-15 05:11:26,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 05:11:26,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36396.86 MB 2025-02-15 05:11:26,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:11:26,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:11:26,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:11:26,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:26,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32447.74 MB 2025-02-15 05:11:26,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33236.43 MB 2025-02-15 05:11:26,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-15 05:11:26,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36213.62 MB 2025-02-15 05:11:26,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36213.62 MB 2025-02-15 05:11:26,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:11:26,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33828.21 MB 2025-02-15 05:11:26,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:11:26,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:11:26,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:11:26,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:26,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33236.43 MB 2025-02-15 05:11:26,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34172.44 MB 2025-02-15 05:11:26,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-15 05:11:26,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36213.62 MB 2025-02-15 05:11:26,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-15 05:11:26,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1774.19 MB 2025-02-15 05:11:26,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36488.32 MB 2025-02-15 05:11:26,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:11:26,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:11:26,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 05:11:26,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:26,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32447.74 MB 2025-02-15 05:11:26,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34172.44 MB 2025-02-15 05:11:26,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-15 05:11:26,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36213.62 MB 2025-02-15 05:11:26,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-15 05:11:26,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1774.19 MB 2025-02-15 05:11:26,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36488.32 MB 2025-02-15 05:11:27,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:11:27,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:11:27,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 05:11:27,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:27,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.69 MB 2025-02-15 05:11:27,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35133.05 MB 2025-02-15 05:11:27,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.35 MB 2025-02-15 05:11:27,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37987.81 MB 2025-02-15 05:11:27,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 05:11:27,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 05:11:27,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35435.78 MB 2025-02-15 05:11:27,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:11:27,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:11:27,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:11:27,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:27,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35305.43 MB 2025-02-15 05:11:27,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35527.97 MB 2025-02-15 05:11:27,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.54 MB 2025-02-15 05:11:27,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 05:11:27,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 05:11:27,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:11:27,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35548.70 MB 2025-02-15 05:11:27,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:11:27,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:11:27,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.97 seconds 2025-02-15 05:11:27,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:27,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30737.84 MB 2025-02-15 05:11:27,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35728.82 MB 2025-02-15 05:11:27,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4990.98 MB 2025-02-15 05:11:27,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56893.64 MB 2025-02-15 05:11:27,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 05:11:27,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18731.76 MB 2025-02-15 05:11:27,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35728.82 MB 2025-02-15 05:11:27,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:11:27,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:11:27,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 05:11:27,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:27,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35728.82 MB 2025-02-15 05:11:27,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34639.34 MB 2025-02-15 05:11:27,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1089.49 MB 2025-02-15 05:11:27,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 05:11:27,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 05:11:27,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:11:27,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36330.96 MB 2025-02-15 05:11:27,404 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 05:11:27,404 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 05:11:27,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:11:27,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:11:27,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:11:27,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:11:27,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34639.34 MB 2025-02-15 05:11:27,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43069.74 MB 2025-02-15 05:11:27,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 05:11:27,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 05:11:27,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46542.09 MB 2025-02-15 05:11:27,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 05:11:27,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43069.74 MB 2025-02-15 05:11:27,664 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 05:11:27,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:27,667 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:11:27,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:27,669 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:11:27,676 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:11:27,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:11:27,678 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:11:27,679 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 05:12:53,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:12:53,362 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:12:53,367 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:12:53,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:12:53,372 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1221, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:12:53,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:12:53,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1221, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:13:12,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:13:12,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:13:12,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.80 seconds 2025-02-15 05:13:12,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:12,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38604.90 MB 2025-02-15 05:13:12,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42925.95 MB 2025-02-15 05:13:12,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4321.05 MB 2025-02-15 05:13:12,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54922.31 MB 2025-02-15 05:13:12,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48513.42 MB 2025-02-15 05:13:12,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6408.90 MB 2025-02-15 05:13:12,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51926.90 MB 2025-02-15 05:13:12,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:13:12,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:13:12,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:13:12,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:12,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42925.95 MB 2025-02-15 05:13:12,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39253.51 MB 2025-02-15 05:13:12,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3672.43 MB 2025-02-15 05:13:12,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 05:13:12,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58479.08 MB 2025-02-15 05:13:12,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9965.67 MB 2025-02-15 05:13:12,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55781.28 MB 2025-02-15 05:13:14,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:13:14,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:13:14,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:13:14,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39253.51 MB 2025-02-15 05:13:14,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39784.36 MB 2025-02-15 05:13:14,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:13:14,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58479.08 MB 2025-02-15 05:13:14,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45606.76 MB 2025-02-15 05:13:14,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12872.32 MB 2025-02-15 05:13:14,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43762.90 MB 2025-02-15 05:13:14,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:13:14,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:13:14,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:13:14,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39784.36 MB 2025-02-15 05:13:14,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41673.89 MB 2025-02-15 05:13:14,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:13:14,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45606.76 MB 2025-02-15 05:13:14,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45606.76 MB 2025-02-15 05:13:14,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:13:14,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43091.32 MB 2025-02-15 05:13:14,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:13:14,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:13:14,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:13:14,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41673.89 MB 2025-02-15 05:13:14,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43915.75 MB 2025-02-15 05:13:14,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:13:14,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45606.76 MB 2025-02-15 05:13:14,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51740.93 MB 2025-02-15 05:13:14,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:13:14,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49460.03 MB 2025-02-15 05:13:14,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:13:14,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:13:14,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:13:14,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39784.36 MB 2025-02-15 05:13:14,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43915.75 MB 2025-02-15 05:13:14,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:13:14,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45606.76 MB 2025-02-15 05:13:14,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51740.93 MB 2025-02-15 05:13:14,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:13:14,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49460.03 MB 2025-02-15 05:13:14,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:13:14,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:13:14,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:13:14,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45449.29 MB 2025-02-15 05:13:14,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46216.29 MB 2025-02-15 05:13:14,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:13:14,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51740.93 MB 2025-02-15 05:13:14,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52160.36 MB 2025-02-15 05:13:14,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:13:14,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46924.08 MB 2025-02-15 05:13:14,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:13:14,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:13:14,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:13:14,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46629.18 MB 2025-02-15 05:13:14,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46858.16 MB 2025-02-15 05:13:14,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-15 05:13:14,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52160.36 MB 2025-02-15 05:13:14,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52160.36 MB 2025-02-15 05:13:14,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:13:14,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47087.22 MB 2025-02-15 05:13:14,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:13:14,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:13:14,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.27 seconds 2025-02-15 05:13:14,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34350.83 MB 2025-02-15 05:13:14,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47059.07 MB 2025-02-15 05:13:14,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12708.23 MB 2025-02-15 05:13:14,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54922.31 MB 2025-02-15 05:13:14,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52160.36 MB 2025-02-15 05:13:14,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2761.95 MB 2025-02-15 05:13:14,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47087.22 MB 2025-02-15 05:13:14,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:13:14,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:13:14,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:13:14,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47059.07 MB 2025-02-15 05:13:14,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39352.56 MB 2025-02-15 05:13:14,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7706.51 MB 2025-02-15 05:13:14,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52160.36 MB 2025-02-15 05:13:14,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52160.36 MB 2025-02-15 05:13:14,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:13:14,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49568.58 MB 2025-02-15 05:13:14,939 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 05:13:14,939 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:13:14,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:13:14,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:13:14,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:13:14,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:13:14,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39352.56 MB 2025-02-15 05:13:14,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47784.02 MB 2025-02-15 05:13:14,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 05:13:14,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52160.36 MB 2025-02-15 05:13:14,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60544.78 MB 2025-02-15 05:13:14,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 05:13:14,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47784.02 MB 2025-02-15 05:13:15,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 05:13:15,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:13:15,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:13:15,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:13:15,176 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:13:15,183 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:13:15,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:13:15,185 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:13:15,185 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:14:01,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:01,575 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:14:01,580 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:14:01,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:01,584 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1955, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:14:01,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:01,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1955, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:14:31,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:14:31,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:14:31,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.30 seconds 2025-02-15 05:14:31,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:31,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43719.52 MB 2025-02-15 05:14:31,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50638.16 MB 2025-02-15 05:14:31,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6918.64 MB 2025-02-15 05:14:31,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68929.19 MB 2025-02-15 05:14:31,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59496.20 MB 2025-02-15 05:14:31,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9432.99 MB 2025-02-15 05:14:31,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59532.95 MB 2025-02-15 05:14:32,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:14:32,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:14:32,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 05:14:32,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:32,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50638.16 MB 2025-02-15 05:14:32,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43069.35 MB 2025-02-15 05:14:32,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7568.81 MB 2025-02-15 05:14:32,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59496.20 MB 2025-02-15 05:14:32,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72041.37 MB 2025-02-15 05:14:32,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12545.16 MB 2025-02-15 05:14:32,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69503.86 MB 2025-02-15 05:14:33,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:14:33,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:14:33,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:14:33,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:33,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43069.35 MB 2025-02-15 05:14:33,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43600.19 MB 2025-02-15 05:14:33,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:14:33,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72041.37 MB 2025-02-15 05:14:33,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52575.60 MB 2025-02-15 05:14:33,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19465.76 MB 2025-02-15 05:14:33,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47578.74 MB 2025-02-15 05:14:33,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:14:33,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:14:33,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:14:33,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:33,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43600.19 MB 2025-02-15 05:14:33,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45489.73 MB 2025-02-15 05:14:33,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:14:33,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52575.60 MB 2025-02-15 05:14:33,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52575.60 MB 2025-02-15 05:14:33,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:14:33,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46907.16 MB 2025-02-15 05:14:34,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:14:34,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:14:34,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:14:34,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45489.73 MB 2025-02-15 05:14:34,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47731.58 MB 2025-02-15 05:14:34,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:14:34,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52575.60 MB 2025-02-15 05:14:34,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56350.47 MB 2025-02-15 05:14:34,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:14:34,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53275.86 MB 2025-02-15 05:14:34,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:14:34,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:14:34,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:14:34,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43600.19 MB 2025-02-15 05:14:34,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47731.58 MB 2025-02-15 05:14:34,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:14:34,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52575.60 MB 2025-02-15 05:14:34,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56350.47 MB 2025-02-15 05:14:34,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:14:34,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53275.86 MB 2025-02-15 05:14:34,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:14:34,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:14:34,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:14:34,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49265.13 MB 2025-02-15 05:14:34,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50032.13 MB 2025-02-15 05:14:34,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:14:34,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56350.47 MB 2025-02-15 05:14:34,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56769.90 MB 2025-02-15 05:14:34,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:14:34,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50739.92 MB 2025-02-15 05:14:34,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:14:34,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:14:34,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:14:34,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50445.02 MB 2025-02-15 05:14:34,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50674.14 MB 2025-02-15 05:14:34,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.12 MB 2025-02-15 05:14:34,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56769.90 MB 2025-02-15 05:14:34,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56769.90 MB 2025-02-15 05:14:34,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:14:34,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50875.93 MB 2025-02-15 05:14:34,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:14:34,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:14:34,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.80 seconds 2025-02-15 05:14:34,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36908.15 MB 2025-02-15 05:14:34,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50874.23 MB 2025-02-15 05:14:34,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13966.08 MB 2025-02-15 05:14:34,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68929.19 MB 2025-02-15 05:14:34,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56769.90 MB 2025-02-15 05:14:34,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12159.29 MB 2025-02-15 05:14:34,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50875.93 MB 2025-02-15 05:14:34,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:14:34,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:14:34,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:14:34,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50874.23 MB 2025-02-15 05:14:34,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41897.30 MB 2025-02-15 05:14:34,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8976.93 MB 2025-02-15 05:14:34,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56769.90 MB 2025-02-15 05:14:34,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56769.90 MB 2025-02-15 05:14:34,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:14:34,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53373.61 MB 2025-02-15 05:14:34,674 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 05:14:34,674 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:14:34,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:14:34,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:14:34,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:14:34,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:14:34,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41897.30 MB 2025-02-15 05:14:34,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50294.70 MB 2025-02-15 05:14:34,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 05:14:34,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56769.90 MB 2025-02-15 05:14:34,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60945.33 MB 2025-02-15 05:14:34,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 05:14:34,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50294.70 MB 2025-02-15 05:14:34,839 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 05:14:34,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:34,841 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:14:34,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:34,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:14:34,847 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:14:34,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:14:34,848 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:14:34,848 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:15:30,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:30,883 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:15:30,888 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:15:30,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:30,892 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1131, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:15:30,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:30,893 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1131, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:15:48,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:15:48,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:15:48,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.47 seconds 2025-02-15 05:15:48,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:48,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37977.76 MB 2025-02-15 05:15:48,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41981.22 MB 2025-02-15 05:15:48,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4003.46 MB 2025-02-15 05:15:48,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69296.19 MB 2025-02-15 05:15:48,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 05:15:48,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21116.22 MB 2025-02-15 05:15:48,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50846.52 MB 2025-02-15 05:15:48,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:15:48,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:15:48,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:15:48,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:48,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41981.22 MB 2025-02-15 05:15:48,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38785.63 MB 2025-02-15 05:15:48,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3195.59 MB 2025-02-15 05:15:48,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 05:15:48,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57470.35 MB 2025-02-15 05:15:48,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9290.38 MB 2025-02-15 05:15:48,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54080.82 MB 2025-02-15 05:15:50,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:15:50,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:15:50,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:15:50,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38785.63 MB 2025-02-15 05:15:50,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39316.47 MB 2025-02-15 05:15:50,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:15:50,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57470.35 MB 2025-02-15 05:15:50,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45592.08 MB 2025-02-15 05:15:50,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11878.27 MB 2025-02-15 05:15:50,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43295.02 MB 2025-02-15 05:15:50,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:15:50,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:15:50,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:15:50,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39316.47 MB 2025-02-15 05:15:50,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41206.01 MB 2025-02-15 05:15:50,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:15:50,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45592.08 MB 2025-02-15 05:15:50,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45592.08 MB 2025-02-15 05:15:50,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:15:50,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42623.44 MB 2025-02-15 05:15:50,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:15:50,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:15:50,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:15:50,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41206.01 MB 2025-02-15 05:15:50,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43447.86 MB 2025-02-15 05:15:50,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:15:50,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45592.08 MB 2025-02-15 05:15:50,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51726.25 MB 2025-02-15 05:15:50,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:15:50,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48992.15 MB 2025-02-15 05:15:50,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:15:50,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:15:50,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:15:50,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39316.47 MB 2025-02-15 05:15:50,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43447.86 MB 2025-02-15 05:15:50,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:15:50,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45592.08 MB 2025-02-15 05:15:50,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51726.25 MB 2025-02-15 05:15:50,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:15:50,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48992.15 MB 2025-02-15 05:15:50,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:15:50,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:15:50,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:15:50,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44981.41 MB 2025-02-15 05:15:50,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45748.41 MB 2025-02-15 05:15:50,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:15:50,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51726.25 MB 2025-02-15 05:15:50,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52145.68 MB 2025-02-15 05:15:50,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:15:50,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46456.20 MB 2025-02-15 05:15:50,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:15:50,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:15:50,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:15:50,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46161.30 MB 2025-02-15 05:15:50,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46389.84 MB 2025-02-15 05:15:50,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-15 05:15:50,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52145.68 MB 2025-02-15 05:15:50,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52145.68 MB 2025-02-15 05:15:50,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:15:50,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46626.33 MB 2025-02-15 05:15:50,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:15:50,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:15:50,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.94 seconds 2025-02-15 05:15:50,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:50,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34037.27 MB 2025-02-15 05:15:50,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46590.30 MB 2025-02-15 05:15:50,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12553.03 MB 2025-02-15 05:15:50,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69296.19 MB 2025-02-15 05:15:50,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52145.68 MB 2025-02-15 05:15:50,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17150.51 MB 2025-02-15 05:15:50,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46626.33 MB 2025-02-15 05:15:51,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:15:51,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:15:51,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:15:51,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:51,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46590.30 MB 2025-02-15 05:15:51,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39032.13 MB 2025-02-15 05:15:51,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7558.17 MB 2025-02-15 05:15:51,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52145.68 MB 2025-02-15 05:15:51,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52145.68 MB 2025-02-15 05:15:51,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:15:51,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49094.29 MB 2025-02-15 05:15:51,122 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 05:15:51,123 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:15:51,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:15:51,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:15:51,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:15:51,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:15:51,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39032.13 MB 2025-02-15 05:15:51,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47445.66 MB 2025-02-15 05:15:51,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-15 05:15:51,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52145.68 MB 2025-02-15 05:15:51,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60509.13 MB 2025-02-15 05:15:51,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 05:15:51,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47445.66 MB 2025-02-15 05:15:51,286 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 05:15:51,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:51,288 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:15:51,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:51,289 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:15:51,293 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:15:51,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:15:51,294 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:15:51,295 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:16:01,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:01,352 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:16:01,356 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:16:01,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:01,360 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1254, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:16:01,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:01,361 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1254, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:16:20,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:16:20,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:16:20,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.47 seconds 2025-02-15 05:16:20,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:20,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38834.85 MB 2025-02-15 05:16:20,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43272.68 MB 2025-02-15 05:16:20,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4437.84 MB 2025-02-15 05:16:20,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68872.57 MB 2025-02-15 05:16:20,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52789.51 MB 2025-02-15 05:16:20,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16083.06 MB 2025-02-15 05:16:20,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52156.59 MB 2025-02-15 05:16:20,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:16:20,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:16:20,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:16:20,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:20,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43272.68 MB 2025-02-15 05:16:20,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39425.07 MB 2025-02-15 05:16:20,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3847.61 MB 2025-02-15 05:16:20,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52789.51 MB 2025-02-15 05:16:20,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61106.81 MB 2025-02-15 05:16:20,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8317.30 MB 2025-02-15 05:16:20,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55812.73 MB 2025-02-15 05:16:22,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:16:22,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:16:22,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:16:22,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:22,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39425.07 MB 2025-02-15 05:16:22,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39955.91 MB 2025-02-15 05:16:22,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:16:22,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61106.81 MB 2025-02-15 05:16:22,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44168.12 MB 2025-02-15 05:16:22,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16938.70 MB 2025-02-15 05:16:22,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43934.46 MB 2025-02-15 05:16:22,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:16:22,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:16:22,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:16:22,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:22,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39955.91 MB 2025-02-15 05:16:22,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41845.45 MB 2025-02-15 05:16:22,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:16:22,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-15 05:16:22,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46055.56 MB 2025-02-15 05:16:22,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 05:16:22,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43262.88 MB 2025-02-15 05:16:23,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:16:23,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:16:23,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:16:23,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41845.45 MB 2025-02-15 05:16:23,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44087.30 MB 2025-02-15 05:16:23,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:16:23,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46055.56 MB 2025-02-15 05:16:23,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51717.87 MB 2025-02-15 05:16:23,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:16:23,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49631.58 MB 2025-02-15 05:16:23,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:16:23,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:16:23,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:16:23,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39955.91 MB 2025-02-15 05:16:23,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44087.30 MB 2025-02-15 05:16:23,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:16:23,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-15 05:16:23,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51717.87 MB 2025-02-15 05:16:23,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 05:16:23,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49631.58 MB 2025-02-15 05:16:23,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:16:23,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:16:23,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:16:23,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45620.85 MB 2025-02-15 05:16:23,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46387.85 MB 2025-02-15 05:16:23,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:16:23,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51717.87 MB 2025-02-15 05:16:23,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52137.30 MB 2025-02-15 05:16:23,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:16:23,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47095.64 MB 2025-02-15 05:16:23,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:16:23,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:16:23,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:16:23,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46800.74 MB 2025-02-15 05:16:23,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47028.35 MB 2025-02-15 05:16:23,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.61 MB 2025-02-15 05:16:23,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52137.30 MB 2025-02-15 05:16:23,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52137.30 MB 2025-02-15 05:16:23,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:16:23,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47237.72 MB 2025-02-15 05:16:23,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:16:23,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:16:23,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.87 seconds 2025-02-15 05:16:23,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34465.81 MB 2025-02-15 05:16:23,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47229.42 MB 2025-02-15 05:16:23,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12763.61 MB 2025-02-15 05:16:23,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68872.57 MB 2025-02-15 05:16:23,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52137.30 MB 2025-02-15 05:16:23,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16735.27 MB 2025-02-15 05:16:23,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47237.72 MB 2025-02-15 05:16:23,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:16:23,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:16:23,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:16:23,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47229.42 MB 2025-02-15 05:16:23,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39470.20 MB 2025-02-15 05:16:23,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7759.22 MB 2025-02-15 05:16:23,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52137.30 MB 2025-02-15 05:16:23,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52137.30 MB 2025-02-15 05:16:23,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:16:23,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49741.09 MB 2025-02-15 05:16:23,517 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:16:23,517 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:16:23,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:16:23,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:16:23,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:16:23,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:16:23,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39470.20 MB 2025-02-15 05:16:23,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47909.22 MB 2025-02-15 05:16:23,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:16:23,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52137.30 MB 2025-02-15 05:16:23,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60528.00 MB 2025-02-15 05:16:23,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:16:23,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47909.22 MB 2025-02-15 05:16:23,681 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:16:23,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:23,682 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:16:23,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:23,683 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:16:23,688 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:16:23,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:16:23,689 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:16:23,689 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:17:30,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:30,491 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:17:30,498 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:17:30,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:30,505 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:17:30,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:30,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:17:32,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:17:32,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:17:32,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.43 seconds 2025-02-15 05:17:32,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:32,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31155.93 MB 2025-02-15 05:17:32,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31693.85 MB 2025-02-15 05:17:32,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-15 05:17:32,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73113.01 MB 2025-02-15 05:17:32,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:17:32,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36358.32 MB 2025-02-15 05:17:32,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40627.30 MB 2025-02-15 05:17:32,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:17:32,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:17:32,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:17:32,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:32,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31693.85 MB 2025-02-15 05:17:32,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31828.06 MB 2025-02-15 05:17:32,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.21 MB 2025-02-15 05:17:32,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:17:32,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:17:32,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:32,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33576.09 MB 2025-02-15 05:17:33,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:17:33,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:17:33,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-15 05:17:33,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31828.06 MB 2025-02-15 05:17:33,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32005.89 MB 2025-02-15 05:17:33,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-15 05:17:33,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:17:33,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36282.83 MB 2025-02-15 05:17:33,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 05:17:33,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35998.75 MB 2025-02-15 05:17:33,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:17:33,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:17:33,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:17:33,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32005.83 MB 2025-02-15 05:17:33,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32638.67 MB 2025-02-15 05:17:33,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-15 05:17:33,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36282.83 MB 2025-02-15 05:17:33,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36282.83 MB 2025-02-15 05:17:33,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:33,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33113.51 MB 2025-02-15 05:17:33,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:17:33,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:17:33,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:17:33,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32638.67 MB 2025-02-15 05:17:33,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33389.73 MB 2025-02-15 05:17:33,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-15 05:17:33,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36282.83 MB 2025-02-15 05:17:33,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36282.83 MB 2025-02-15 05:17:33,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:33,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35247.02 MB 2025-02-15 05:17:33,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:17:33,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:17:33,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:17:33,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32005.83 MB 2025-02-15 05:17:33,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33389.73 MB 2025-02-15 05:17:33,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-15 05:17:33,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36282.83 MB 2025-02-15 05:17:33,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36282.83 MB 2025-02-15 05:17:33,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:33,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35247.02 MB 2025-02-15 05:17:33,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:17:33,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:17:33,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:17:33,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33903.47 MB 2025-02-15 05:17:33,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34160.41 MB 2025-02-15 05:17:33,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-15 05:17:33,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36282.83 MB 2025-02-15 05:17:33,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36423.34 MB 2025-02-15 05:17:33,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 140.51 MB 2025-02-15 05:17:33,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34408.76 MB 2025-02-15 05:17:33,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:17:33,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:17:33,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:17:33,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34298.74 MB 2025-02-15 05:17:33,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34517.80 MB 2025-02-15 05:17:33,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.06 MB 2025-02-15 05:17:33,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36423.34 MB 2025-02-15 05:17:33,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36423.34 MB 2025-02-15 05:17:33,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:33,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34517.80 MB 2025-02-15 05:17:33,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:17:33,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:17:33,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-15 05:17:33,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:33,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30626.35 MB 2025-02-15 05:17:33,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31361.30 MB 2025-02-15 05:17:33,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.95 MB 2025-02-15 05:17:33,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73113.01 MB 2025-02-15 05:17:33,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36423.34 MB 2025-02-15 05:17:33,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36689.67 MB 2025-02-15 05:17:33,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34718.72 MB 2025-02-15 05:17:34,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:17:34,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:17:34,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:17:34,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:34,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31361.30 MB 2025-02-15 05:17:34,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34373.12 MB 2025-02-15 05:17:34,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.82 MB 2025-02-15 05:17:34,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36423.34 MB 2025-02-15 05:17:34,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36423.34 MB 2025-02-15 05:17:34,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:17:34,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34674.27 MB 2025-02-15 05:17:34,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 05:17:34,050 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:17:34,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:17:34,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:17:34,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:17:34,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:17:34,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34373.12 MB 2025-02-15 05:17:34,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42806.42 MB 2025-02-15 05:17:34,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 05:17:34,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36423.34 MB 2025-02-15 05:17:34,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46904.90 MB 2025-02-15 05:17:34,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 05:17:34,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42806.42 MB 2025-02-15 05:17:34,217 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 05:17:34,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:34,218 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:17:34,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:34,219 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:17:34,224 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:17:34,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:17:34,225 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:17:34,225 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:18:35,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:35,587 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:18:35,592 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:18:35,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:35,595 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1341, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:18:35,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:35,596 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1341, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:18:56,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:18:56,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:18:56,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.55 seconds 2025-02-15 05:18:56,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:56,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39441.07 MB 2025-02-15 05:18:56,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44186.93 MB 2025-02-15 05:18:56,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4745.85 MB 2025-02-15 05:18:56,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55289.32 MB 2025-02-15 05:18:56,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53133.44 MB 2025-02-15 05:18:56,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2155.87 MB 2025-02-15 05:18:56,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52989.31 MB 2025-02-15 05:18:56,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:18:56,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:18:56,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:18:56,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:56,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44186.93 MB 2025-02-15 05:18:56,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39877.36 MB 2025-02-15 05:18:56,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4309.57 MB 2025-02-15 05:18:56,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53133.44 MB 2025-02-15 05:18:56,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62212.01 MB 2025-02-15 05:18:56,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9078.57 MB 2025-02-15 05:18:56,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57834.61 MB 2025-02-15 05:18:58,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:18:58,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:18:58,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:18:58,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39877.36 MB 2025-02-15 05:18:58,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40408.20 MB 2025-02-15 05:18:58,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:18:58,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62212.01 MB 2025-02-15 05:18:58,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48387.59 MB 2025-02-15 05:18:58,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13824.43 MB 2025-02-15 05:18:58,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44386.75 MB 2025-02-15 05:18:58,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:18:58,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:18:58,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:18:58,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40408.20 MB 2025-02-15 05:18:58,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42297.73 MB 2025-02-15 05:18:58,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:18:58,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48387.59 MB 2025-02-15 05:18:58,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48387.59 MB 2025-02-15 05:18:58,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:18:58,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43715.16 MB 2025-02-15 05:18:58,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:18:58,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:18:58,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:18:58,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42297.73 MB 2025-02-15 05:18:58,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44539.59 MB 2025-02-15 05:18:58,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:18:58,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48387.59 MB 2025-02-15 05:18:58,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52634.32 MB 2025-02-15 05:18:58,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 05:18:58,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50083.87 MB 2025-02-15 05:18:58,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:18:58,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:18:58,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:18:58,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40408.20 MB 2025-02-15 05:18:58,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44539.59 MB 2025-02-15 05:18:58,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:18:58,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48387.59 MB 2025-02-15 05:18:58,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52634.32 MB 2025-02-15 05:18:58,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 05:18:58,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50083.87 MB 2025-02-15 05:18:58,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:18:58,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:18:58,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:18:58,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46073.13 MB 2025-02-15 05:18:58,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46840.13 MB 2025-02-15 05:18:58,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:18:58,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52634.32 MB 2025-02-15 05:18:58,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53053.75 MB 2025-02-15 05:18:58,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:18:58,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47547.92 MB 2025-02-15 05:18:58,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:18:58,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:18:58,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:18:58,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47253.02 MB 2025-02-15 05:18:58,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47480.31 MB 2025-02-15 05:18:58,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.29 MB 2025-02-15 05:18:58,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53053.75 MB 2025-02-15 05:18:58,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53053.75 MB 2025-02-15 05:18:58,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:18:58,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47691.18 MB 2025-02-15 05:18:58,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:18:58,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:18:58,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.96 seconds 2025-02-15 05:18:58,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34768.92 MB 2025-02-15 05:18:58,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47680.67 MB 2025-02-15 05:18:58,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12911.75 MB 2025-02-15 05:18:58,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55289.32 MB 2025-02-15 05:18:58,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53053.75 MB 2025-02-15 05:18:58,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2235.56 MB 2025-02-15 05:18:58,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47691.18 MB 2025-02-15 05:18:58,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:18:58,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:18:58,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:18:58,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47680.67 MB 2025-02-15 05:18:58,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39762.27 MB 2025-02-15 05:18:58,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7918.41 MB 2025-02-15 05:18:58,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53053.75 MB 2025-02-15 05:18:58,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53053.75 MB 2025-02-15 05:18:58,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:18:58,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50183.43 MB 2025-02-15 05:18:58,844 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 05:18:58,844 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:18:58,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:18:58,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:18:58,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:18:58,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:18:58,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39762.27 MB 2025-02-15 05:18:58,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48171.55 MB 2025-02-15 05:18:58,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.29 MB 2025-02-15 05:18:58,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53053.75 MB 2025-02-15 05:18:58,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57233.38 MB 2025-02-15 05:18:58,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 05:18:58,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48171.55 MB 2025-02-15 05:18:59,008 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 05:18:59,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:59,010 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:18:59,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:59,011 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:18:59,015 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:18:59,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:18:59,016 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:18:59,016 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:19:06,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:06,686 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:19:06,690 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:19:06,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:06,694 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1444, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:19:06,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:06,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1444, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:19:29,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:19:29,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:19:29,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.44 seconds 2025-02-15 05:19:29,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:29,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40158.80 MB 2025-02-15 05:19:29,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45269.55 MB 2025-02-15 05:19:29,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5110.76 MB 2025-02-15 05:19:29,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65592.62 MB 2025-02-15 05:19:29,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53460.60 MB 2025-02-15 05:19:29,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12132.02 MB 2025-02-15 05:19:29,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54160.28 MB 2025-02-15 05:19:29,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:19:29,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:19:29,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:19:29,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:29,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45269.55 MB 2025-02-15 05:19:29,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40412.82 MB 2025-02-15 05:19:29,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4856.73 MB 2025-02-15 05:19:29,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53460.60 MB 2025-02-15 05:19:29,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63132.66 MB 2025-02-15 05:19:29,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9672.07 MB 2025-02-15 05:19:29,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59816.89 MB 2025-02-15 05:19:31,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:19:31,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:19:31,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:19:31,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40412.82 MB 2025-02-15 05:19:31,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40943.66 MB 2025-02-15 05:19:31,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:19:31,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63132.66 MB 2025-02-15 05:19:31,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44170.22 MB 2025-02-15 05:19:31,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18962.45 MB 2025-02-15 05:19:31,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44922.21 MB 2025-02-15 05:19:31,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:19:31,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:19:31,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:19:31,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40943.66 MB 2025-02-15 05:19:31,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42833.20 MB 2025-02-15 05:19:31,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:19:31,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44170.22 MB 2025-02-15 05:19:31,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47001.37 MB 2025-02-15 05:19:31,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:19:31,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44250.63 MB 2025-02-15 05:19:31,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:19:31,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:19:31,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:19:31,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42833.20 MB 2025-02-15 05:19:31,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45075.05 MB 2025-02-15 05:19:31,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:19:31,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47001.37 MB 2025-02-15 05:19:31,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52663.68 MB 2025-02-15 05:19:31,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:19:31,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50619.34 MB 2025-02-15 05:19:31,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:19:31,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:19:31,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:19:31,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40943.66 MB 2025-02-15 05:19:31,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45075.05 MB 2025-02-15 05:19:31,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:19:31,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44170.22 MB 2025-02-15 05:19:31,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52663.68 MB 2025-02-15 05:19:31,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 05:19:31,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50619.34 MB 2025-02-15 05:19:31,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:19:31,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:19:31,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:19:31,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46608.60 MB 2025-02-15 05:19:31,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47375.60 MB 2025-02-15 05:19:31,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:19:31,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52663.68 MB 2025-02-15 05:19:31,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 05:19:31,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:19:31,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48083.39 MB 2025-02-15 05:19:31,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:19:31,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:19:31,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:19:31,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47788.49 MB 2025-02-15 05:19:31,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48016.64 MB 2025-02-15 05:19:31,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-15 05:19:31,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 05:19:31,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 05:19:31,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:31,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48243.67 MB 2025-02-15 05:19:31,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:19:31,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:19:31,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.88 seconds 2025-02-15 05:19:31,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35127.78 MB 2025-02-15 05:19:31,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48216.70 MB 2025-02-15 05:19:31,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13088.92 MB 2025-02-15 05:19:31,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65592.62 MB 2025-02-15 05:19:31,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 05:19:31,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12509.51 MB 2025-02-15 05:19:31,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48243.67 MB 2025-02-15 05:19:31,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:19:31,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:19:31,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:19:31,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48216.70 MB 2025-02-15 05:19:31,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40116.55 MB 2025-02-15 05:19:31,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8100.15 MB 2025-02-15 05:19:31,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 05:19:31,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 05:19:31,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:31,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50715.77 MB 2025-02-15 05:19:31,864 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 05:19:31,865 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:19:31,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:19:31,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:19:31,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:19:31,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:31,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40116.55 MB 2025-02-15 05:19:31,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48512.93 MB 2025-02-15 05:19:31,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.37 MB 2025-02-15 05:19:31,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 05:19:31,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57258.54 MB 2025-02-15 05:19:31,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 05:19:31,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48512.93 MB 2025-02-15 05:19:32,030 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 05:19:32,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:32,031 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:19:32,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:32,032 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:19:32,037 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:19:32,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:32,038 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:19:32,038 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:19:42,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:42,840 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:19:42,844 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:19:42,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:42,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:19:42,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:42,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:19:45,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:19:45,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:19:45,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.41 seconds 2025-02-15 05:19:45,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:45,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31162.90 MB 2025-02-15 05:19:45,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31704.36 MB 2025-02-15 05:19:45,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 05:19:45,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65605.21 MB 2025-02-15 05:19:45,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:45,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28850.52 MB 2025-02-15 05:19:45,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40634.27 MB 2025-02-15 05:19:45,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:19:45,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:19:45,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:19:45,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:45,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31704.36 MB 2025-02-15 05:19:45,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31840.28 MB 2025-02-15 05:19:45,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.92 MB 2025-02-15 05:19:45,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:45,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:45,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:45,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33600.64 MB 2025-02-15 05:19:45,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:19:45,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:19:45,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-15 05:19:45,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:45,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31840.28 MB 2025-02-15 05:19:45,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32019.44 MB 2025-02-15 05:19:45,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 179.16 MB 2025-02-15 05:19:45,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:45,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:45,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:45,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36010.97 MB 2025-02-15 05:19:45,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:19:45,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:19:45,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:19:45,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:45,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32019.37 MB 2025-02-15 05:19:45,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32656.94 MB 2025-02-15 05:19:45,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.56 MB 2025-02-15 05:19:45,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:45,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:45,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:45,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33135.32 MB 2025-02-15 05:19:46,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:19:46,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:19:46,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:19:46,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32656.94 MB 2025-02-15 05:19:46,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33413.61 MB 2025-02-15 05:19:46,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 756.67 MB 2025-02-15 05:19:46,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:46,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:46,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:46,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35284.76 MB 2025-02-15 05:19:46,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:19:46,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:19:46,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:19:46,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32019.37 MB 2025-02-15 05:19:46,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33413.61 MB 2025-02-15 05:19:46,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1394.23 MB 2025-02-15 05:19:46,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:46,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36754.69 MB 2025-02-15 05:19:46,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:46,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35284.76 MB 2025-02-15 05:19:46,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:19:46,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:19:46,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:19:46,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33931.18 MB 2025-02-15 05:19:46,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34190.04 MB 2025-02-15 05:19:46,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.86 MB 2025-02-15 05:19:46,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36754.69 MB 2025-02-15 05:19:46,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36895.20 MB 2025-02-15 05:19:46,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 140.51 MB 2025-02-15 05:19:46,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34440.60 MB 2025-02-15 05:19:46,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:19:46,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:19:46,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:19:46,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34329.40 MB 2025-02-15 05:19:46,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34551.62 MB 2025-02-15 05:19:46,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.23 MB 2025-02-15 05:19:46,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36895.20 MB 2025-02-15 05:19:46,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36895.20 MB 2025-02-15 05:19:46,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:46,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34551.62 MB 2025-02-15 05:19:46,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:19:46,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:19:46,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.23 seconds 2025-02-15 05:19:46,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30629.84 MB 2025-02-15 05:19:46,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31368.90 MB 2025-02-15 05:19:46,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.07 MB 2025-02-15 05:19:46,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65605.21 MB 2025-02-15 05:19:46,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36895.20 MB 2025-02-15 05:19:46,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28710.01 MB 2025-02-15 05:19:46,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34751.34 MB 2025-02-15 05:19:46,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:19:46,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:19:46,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:19:46,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31368.90 MB 2025-02-15 05:19:46,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34362.83 MB 2025-02-15 05:19:46,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2993.93 MB 2025-02-15 05:19:46,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36895.20 MB 2025-02-15 05:19:46,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36895.20 MB 2025-02-15 05:19:46,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:19:46,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34662.17 MB 2025-02-15 05:19:46,365 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8107, cut from 8109 2025-02-15 05:19:46,365 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 05:19:46,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:19:46,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:19:46,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:19:46,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:19:46,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34362.83 MB 2025-02-15 05:19:46,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42745.52 MB 2025-02-15 05:19:46,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8382.69 MB 2025-02-15 05:19:46,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36895.20 MB 2025-02-15 05:19:46,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47313.85 MB 2025-02-15 05:19:46,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10418.65 MB 2025-02-15 05:19:46,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42745.52 MB 2025-02-15 05:19:46,528 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7899] 2025-02-15 05:19:46,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:46,530 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:19:46,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:46,531 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:19:46,535 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:19:46,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:19:46,537 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:19:46,537 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 05:20:38,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:38,202 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:20:38,207 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:20:38,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:38,211 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:20:38,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:38,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:20:40,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:20:40,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:20:40,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.44 seconds 2025-02-15 05:20:40,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:40,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36581.97 MB 2025-02-15 05:20:40,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37141.13 MB 2025-02-15 05:20:40,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-15 05:20:40,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55647.93 MB 2025-02-15 05:20:40,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-15 05:20:40,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14726.20 MB 2025-02-15 05:20:40,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46054.15 MB 2025-02-15 05:20:40,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:20:40,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:20:40,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:20:40,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:40,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37141.13 MB 2025-02-15 05:20:40,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37152.18 MB 2025-02-15 05:20:40,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11.06 MB 2025-02-15 05:20:40,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-15 05:20:40,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-15 05:20:40,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:20:40,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38840.76 MB 2025-02-15 05:20:41,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:20:41,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:20:41,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.58 seconds 2025-02-15 05:20:41,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37152.18 MB 2025-02-15 05:20:41,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37312.76 MB 2025-02-15 05:20:41,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 160.58 MB 2025-02-15 05:20:41,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-15 05:20:41,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-15 05:20:41,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:20:41,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41236.90 MB 2025-02-15 05:20:41,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:20:41,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:20:41,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:20:41,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37312.70 MB 2025-02-15 05:20:41,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37884.14 MB 2025-02-15 05:20:41,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 571.45 MB 2025-02-15 05:20:41,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-15 05:20:41,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-15 05:20:41,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:20:41,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38313.55 MB 2025-02-15 05:20:41,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:20:41,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:20:41,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:20:41,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37884.14 MB 2025-02-15 05:20:41,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38579.29 MB 2025-02-15 05:20:41,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 695.15 MB 2025-02-15 05:20:41,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-15 05:20:41,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41783.66 MB 2025-02-15 05:20:41,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 861.93 MB 2025-02-15 05:20:41,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40240.53 MB 2025-02-15 05:20:41,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:20:41,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:20:41,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 05:20:41,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37312.70 MB 2025-02-15 05:20:41,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38579.29 MB 2025-02-15 05:20:41,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1266.59 MB 2025-02-15 05:20:41,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-15 05:20:41,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41783.66 MB 2025-02-15 05:20:41,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 861.93 MB 2025-02-15 05:20:41,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40240.53 MB 2025-02-15 05:20:41,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:20:41,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:20:41,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:20:41,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39249.75 MB 2025-02-15 05:20:41,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39541.24 MB 2025-02-15 05:20:41,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.49 MB 2025-02-15 05:20:41,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41783.66 MB 2025-02-15 05:20:41,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41972.40 MB 2025-02-15 05:20:41,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 188.74 MB 2025-02-15 05:20:41,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39755.35 MB 2025-02-15 05:20:41,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:20:41,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:20:41,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:20:41,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39725.62 MB 2025-02-15 05:20:41,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39957.72 MB 2025-02-15 05:20:41,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.10 MB 2025-02-15 05:20:41,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41972.40 MB 2025-02-15 05:20:41,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41972.40 MB 2025-02-15 05:20:41,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:20:41,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39957.72 MB 2025-02-15 05:20:41,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:20:41,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:20:41,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-15 05:20:41,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36031.49 MB 2025-02-15 05:20:41,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40158.33 MB 2025-02-15 05:20:41,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4126.84 MB 2025-02-15 05:20:41,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55647.93 MB 2025-02-15 05:20:41,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41972.40 MB 2025-02-15 05:20:41,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13675.53 MB 2025-02-15 05:20:41,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40158.33 MB 2025-02-15 05:20:41,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:20:41,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:20:41,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:20:41,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40158.33 MB 2025-02-15 05:20:41,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34438.56 MB 2025-02-15 05:20:41,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5719.77 MB 2025-02-15 05:20:41,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41972.40 MB 2025-02-15 05:20:41,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44119.88 MB 2025-02-15 05:20:41,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2147.48 MB 2025-02-15 05:20:41,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42664.73 MB 2025-02-15 05:20:41,744 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 05:20:41,745 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:20:41,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:20:41,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:20:41,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:20:41,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:20:41,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34438.56 MB 2025-02-15 05:20:41,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42857.64 MB 2025-02-15 05:20:41,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 05:20:41,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44119.88 MB 2025-02-15 05:20:41,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52491.71 MB 2025-02-15 05:20:41,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 05:20:41,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42857.64 MB 2025-02-15 05:20:41,912 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 05:20:41,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:41,914 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:20:41,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:41,915 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:20:41,919 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:20:41,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:20:41,920 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:20:41,921 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:21:47,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:21:47,707 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:21:47,716 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:21:47,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:21:47,725 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1328, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:21:47,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:21:47,727 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1328, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:22:08,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:22:08,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:22:08,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.45 seconds 2025-02-15 05:22:08,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:08,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39350.49 MB 2025-02-15 05:22:08,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44050.21 MB 2025-02-15 05:22:08,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4699.72 MB 2025-02-15 05:22:08,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60863.55 MB 2025-02-15 05:22:08,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53049.56 MB 2025-02-15 05:22:08,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7813.99 MB 2025-02-15 05:22:08,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52898.72 MB 2025-02-15 05:22:08,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:22:08,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:22:08,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:22:08,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:08,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44050.21 MB 2025-02-15 05:22:08,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39809.77 MB 2025-02-15 05:22:08,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4240.43 MB 2025-02-15 05:22:08,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53049.56 MB 2025-02-15 05:22:08,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60565.75 MB 2025-02-15 05:22:08,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7516.19 MB 2025-02-15 05:22:08,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55791.18 MB 2025-02-15 05:22:10,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:22:10,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:22:10,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:22:10,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39809.77 MB 2025-02-15 05:22:10,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40340.62 MB 2025-02-15 05:22:10,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:22:10,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60565.75 MB 2025-02-15 05:22:10,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48349.84 MB 2025-02-15 05:22:10,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12215.91 MB 2025-02-15 05:22:10,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44319.16 MB 2025-02-15 05:22:10,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:22:10,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:22:10,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:22:10,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40340.62 MB 2025-02-15 05:22:10,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42230.15 MB 2025-02-15 05:22:10,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:22:10,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48349.84 MB 2025-02-15 05:22:10,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48349.84 MB 2025-02-15 05:22:10,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:22:10,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43647.58 MB 2025-02-15 05:22:10,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:22:10,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:22:10,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:22:10,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42230.15 MB 2025-02-15 05:22:10,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44472.01 MB 2025-02-15 05:22:10,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:22:10,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48349.84 MB 2025-02-15 05:22:10,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53068.43 MB 2025-02-15 05:22:10,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:22:10,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50016.29 MB 2025-02-15 05:22:10,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:22:10,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:22:10,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:22:10,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40340.62 MB 2025-02-15 05:22:10,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44472.01 MB 2025-02-15 05:22:10,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:22:10,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48349.84 MB 2025-02-15 05:22:10,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53068.43 MB 2025-02-15 05:22:10,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:22:10,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50016.29 MB 2025-02-15 05:22:10,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:22:10,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:22:10,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:22:10,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46005.55 MB 2025-02-15 05:22:10,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46772.55 MB 2025-02-15 05:22:10,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:22:10,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53068.43 MB 2025-02-15 05:22:10,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53487.86 MB 2025-02-15 05:22:10,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:22:10,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47480.34 MB 2025-02-15 05:22:10,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:22:10,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:22:10,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:22:10,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47185.44 MB 2025-02-15 05:22:10,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47412.41 MB 2025-02-15 05:22:10,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.97 MB 2025-02-15 05:22:10,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53487.86 MB 2025-02-15 05:22:10,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53487.86 MB 2025-02-15 05:22:10,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:22:10,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47645.08 MB 2025-02-15 05:22:10,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:22:10,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:22:10,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.91 seconds 2025-02-15 05:22:10,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34723.63 MB 2025-02-15 05:22:10,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47612.97 MB 2025-02-15 05:22:10,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12889.34 MB 2025-02-15 05:22:10,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60863.55 MB 2025-02-15 05:22:10,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53487.86 MB 2025-02-15 05:22:10,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7375.68 MB 2025-02-15 05:22:10,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47645.08 MB 2025-02-15 05:22:10,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:22:10,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:22:10,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:22:10,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47612.97 MB 2025-02-15 05:22:10,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39720.79 MB 2025-02-15 05:22:10,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7892.18 MB 2025-02-15 05:22:10,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53487.86 MB 2025-02-15 05:22:10,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53487.86 MB 2025-02-15 05:22:10,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:22:10,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50118.95 MB 2025-02-15 05:22:10,932 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 05:22:10,932 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:22:10,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:22:10,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:22:10,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:22:10,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:22:10,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39720.79 MB 2025-02-15 05:22:10,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48138.42 MB 2025-02-15 05:22:10,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.63 MB 2025-02-15 05:22:10,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53487.86 MB 2025-02-15 05:22:10,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57671.68 MB 2025-02-15 05:22:10,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 05:22:10,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48138.42 MB 2025-02-15 05:22:11,096 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 05:22:11,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:22:11,098 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:22:11,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:22:11,099 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:22:11,103 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:22:11,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:22:11,104 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:22:11,104 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:23:02,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:02,584 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:23:02,589 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:23:02,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:02,593 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1552, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:23:02,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:02,594 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1552, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:23:26,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:23:26,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:23:26,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.94 seconds 2025-02-15 05:23:26,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:26,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40911.36 MB 2025-02-15 05:23:26,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46403.80 MB 2025-02-15 05:23:26,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5492.44 MB 2025-02-15 05:23:26,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66039.32 MB 2025-02-15 05:23:26,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53838.09 MB 2025-02-15 05:23:26,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12201.23 MB 2025-02-15 05:23:26,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55366.37 MB 2025-02-15 05:23:26,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:23:26,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:23:26,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:23:26,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:26,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46403.80 MB 2025-02-15 05:23:26,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40974.28 MB 2025-02-15 05:23:26,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5429.52 MB 2025-02-15 05:23:26,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53838.09 MB 2025-02-15 05:23:26,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63724.06 MB 2025-02-15 05:23:26,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9885.97 MB 2025-02-15 05:23:26,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61264.85 MB 2025-02-15 05:23:28,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:23:28,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:23:28,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:23:28,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:28,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40974.28 MB 2025-02-15 05:23:28,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41505.12 MB 2025-02-15 05:23:28,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:23:28,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63724.06 MB 2025-02-15 05:23:28,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44178.60 MB 2025-02-15 05:23:28,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19545.46 MB 2025-02-15 05:23:28,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45483.67 MB 2025-02-15 05:23:28,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:23:28,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:23:28,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:23:28,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:28,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41505.12 MB 2025-02-15 05:23:28,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43394.66 MB 2025-02-15 05:23:28,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:23:28,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44178.60 MB 2025-02-15 05:23:28,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47009.76 MB 2025-02-15 05:23:28,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:23:28,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44812.08 MB 2025-02-15 05:23:28,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:23:28,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:23:28,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:23:28,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:28,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43394.66 MB 2025-02-15 05:23:28,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45636.51 MB 2025-02-15 05:23:28,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:23:28,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47009.76 MB 2025-02-15 05:23:28,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53380.91 MB 2025-02-15 05:23:28,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 05:23:28,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51181.84 MB 2025-02-15 05:23:28,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:23:28,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:23:28,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:23:28,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:28,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41505.12 MB 2025-02-15 05:23:28,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45636.51 MB 2025-02-15 05:23:28,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:23:28,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44178.60 MB 2025-02-15 05:23:28,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53380.91 MB 2025-02-15 05:23:28,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-15 05:23:28,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51181.84 MB 2025-02-15 05:23:28,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:23:28,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:23:28,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:23:28,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:28,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47170.05 MB 2025-02-15 05:23:28,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47937.06 MB 2025-02-15 05:23:28,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:23:28,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53380.91 MB 2025-02-15 05:23:28,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53800.34 MB 2025-02-15 05:23:28,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:23:28,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48644.84 MB 2025-02-15 05:23:29,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:23:29,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:23:29,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:23:29,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:29,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48349.94 MB 2025-02-15 05:23:29,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48580.33 MB 2025-02-15 05:23:29,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.39 MB 2025-02-15 05:23:29,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53800.34 MB 2025-02-15 05:23:29,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53800.34 MB 2025-02-15 05:23:29,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:23:29,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48780.86 MB 2025-02-15 05:23:29,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:23:29,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:23:29,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.41 seconds 2025-02-15 05:23:29,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:29,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35504.06 MB 2025-02-15 05:23:29,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48781.40 MB 2025-02-15 05:23:29,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13277.34 MB 2025-02-15 05:23:29,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66039.32 MB 2025-02-15 05:23:29,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53800.34 MB 2025-02-15 05:23:29,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12238.98 MB 2025-02-15 05:23:29,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48781.40 MB 2025-02-15 05:23:29,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:23:29,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:23:29,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:23:29,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:29,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48781.40 MB 2025-02-15 05:23:29,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40508.45 MB 2025-02-15 05:23:29,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8272.95 MB 2025-02-15 05:23:29,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53800.34 MB 2025-02-15 05:23:29,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53800.34 MB 2025-02-15 05:23:29,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:23:29,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51293.07 MB 2025-02-15 05:23:29,294 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:23:29,294 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:23:29,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:23:29,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:23:29,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:23:29,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:23:29,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40508.45 MB 2025-02-15 05:23:29,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48947.48 MB 2025-02-15 05:23:29,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:23:29,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53800.34 MB 2025-02-15 05:23:29,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62191.04 MB 2025-02-15 05:23:29,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:23:29,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48947.48 MB 2025-02-15 05:23:29,459 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:23:29,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:29,460 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:23:29,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:29,461 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:23:29,466 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:23:29,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:23:29,467 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:23:29,467 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:24:20,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:20,306 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:24:20,311 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:24:20,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:20,314 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1538, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:24:20,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:20,315 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1538, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:24:44,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:24:44,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:24:44,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.78 seconds 2025-02-15 05:24:44,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:44,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40813.80 MB 2025-02-15 05:24:44,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46256.70 MB 2025-02-15 05:24:44,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5442.90 MB 2025-02-15 05:24:44,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74776.05 MB 2025-02-15 05:24:44,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53817.11 MB 2025-02-15 05:24:44,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20958.94 MB 2025-02-15 05:24:44,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55269.47 MB 2025-02-15 05:24:44,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:24:44,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:24:44,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:24:44,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:44,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46256.70 MB 2025-02-15 05:24:44,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40901.50 MB 2025-02-15 05:24:44,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5355.20 MB 2025-02-15 05:24:44,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53817.11 MB 2025-02-15 05:24:44,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61475.91 MB 2025-02-15 05:24:44,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7658.80 MB 2025-02-15 05:24:44,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58137.23 MB 2025-02-15 05:24:46,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:24:46,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:24:46,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:24:46,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40901.50 MB 2025-02-15 05:24:46,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41432.34 MB 2025-02-15 05:24:46,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:24:46,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61475.91 MB 2025-02-15 05:24:46,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48372.91 MB 2025-02-15 05:24:46,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13103.01 MB 2025-02-15 05:24:46,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45410.99 MB 2025-02-15 05:24:46,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:24:46,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:24:46,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:24:46,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41432.34 MB 2025-02-15 05:24:46,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43321.87 MB 2025-02-15 05:24:46,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:24:46,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48372.91 MB 2025-02-15 05:24:46,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48372.91 MB 2025-02-15 05:24:46,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:24:46,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44739.30 MB 2025-02-15 05:24:46,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:24:46,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:24:46,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:24:46,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43321.87 MB 2025-02-15 05:24:46,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45563.73 MB 2025-02-15 05:24:46,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:24:46,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48372.91 MB 2025-02-15 05:24:46,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53563.36 MB 2025-02-15 05:24:46,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 05:24:46,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51108.01 MB 2025-02-15 05:24:46,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:24:46,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:24:46,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:24:46,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41432.34 MB 2025-02-15 05:24:46,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45563.73 MB 2025-02-15 05:24:46,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:24:46,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48372.91 MB 2025-02-15 05:24:46,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53563.36 MB 2025-02-15 05:24:46,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 05:24:46,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51108.01 MB 2025-02-15 05:24:46,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:24:46,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:24:46,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 05:24:46,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47097.27 MB 2025-02-15 05:24:46,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47864.27 MB 2025-02-15 05:24:46,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:24:46,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53563.36 MB 2025-02-15 05:24:46,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53982.79 MB 2025-02-15 05:24:46,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:24:46,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48572.06 MB 2025-02-15 05:24:46,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:24:46,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:24:46,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:24:46,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48277.16 MB 2025-02-15 05:24:46,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48505.53 MB 2025-02-15 05:24:46,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 05:24:46,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53982.79 MB 2025-02-15 05:24:46,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53982.79 MB 2025-02-15 05:24:46,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:24:46,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48744.80 MB 2025-02-15 05:24:46,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:24:46,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:24:46,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.29 seconds 2025-02-15 05:24:46,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35455.29 MB 2025-02-15 05:24:46,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48705.82 MB 2025-02-15 05:24:46,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13250.53 MB 2025-02-15 05:24:46,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74776.05 MB 2025-02-15 05:24:46,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53982.79 MB 2025-02-15 05:24:46,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20793.26 MB 2025-02-15 05:24:46,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48744.80 MB 2025-02-15 05:24:46,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:24:46,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:24:46,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:24:46,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48705.82 MB 2025-02-15 05:24:46,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40447.49 MB 2025-02-15 05:24:46,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8258.33 MB 2025-02-15 05:24:46,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53982.79 MB 2025-02-15 05:24:46,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53982.79 MB 2025-02-15 05:24:46,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:24:46,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51207.66 MB 2025-02-15 05:24:46,894 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 05:24:46,894 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:24:46,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:24:46,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:24:46,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:24:46,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:24:46,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40447.49 MB 2025-02-15 05:24:46,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48853.15 MB 2025-02-15 05:24:46,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 05:24:46,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53982.79 MB 2025-02-15 05:24:46,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58162.41 MB 2025-02-15 05:24:46,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 05:24:46,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48853.15 MB 2025-02-15 05:24:47,059 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 05:24:47,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:47,060 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:24:47,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:47,061 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:24:47,066 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:24:47,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:24:47,067 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:24:47,067 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:25:46,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:25:46,922 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:25:46,931 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:25:46,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:25:46,938 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1097, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:25:46,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:25:46,940 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1097, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:26:03,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:26:03,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:26:03,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.03 seconds 2025-02-15 05:26:03,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:03,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37740.84 MB 2025-02-15 05:26:03,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41623.07 MB 2025-02-15 05:26:03,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3882.22 MB 2025-02-15 05:26:03,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66521.66 MB 2025-02-15 05:26:03,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48054.14 MB 2025-02-15 05:26:03,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18467.52 MB 2025-02-15 05:26:03,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50609.86 MB 2025-02-15 05:26:04,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:26:04,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:26:04,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:26:04,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:04,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41623.07 MB 2025-02-15 05:26:04,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38608.88 MB 2025-02-15 05:26:04,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3014.19 MB 2025-02-15 05:26:04,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48054.14 MB 2025-02-15 05:26:04,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56711.18 MB 2025-02-15 05:26:04,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8657.04 MB 2025-02-15 05:26:04,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53089.32 MB 2025-02-15 05:26:06,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:26:06,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:26:06,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:26:06,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38608.88 MB 2025-02-15 05:26:06,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39139.72 MB 2025-02-15 05:26:06,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:26:06,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56711.18 MB 2025-02-15 05:26:06,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45585.79 MB 2025-02-15 05:26:06,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11125.39 MB 2025-02-15 05:26:06,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43118.27 MB 2025-02-15 05:26:06,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:26:06,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:26:06,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:26:06,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39139.72 MB 2025-02-15 05:26:06,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41029.25 MB 2025-02-15 05:26:06,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:26:06,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45585.79 MB 2025-02-15 05:26:06,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45585.79 MB 2025-02-15 05:26:06,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:26:06,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42446.68 MB 2025-02-15 05:26:06,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:26:06,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:26:06,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:26:06,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41029.25 MB 2025-02-15 05:26:06,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43271.11 MB 2025-02-15 05:26:06,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:26:06,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45585.79 MB 2025-02-15 05:26:06,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51248.10 MB 2025-02-15 05:26:06,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:26:06,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48815.39 MB 2025-02-15 05:26:06,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:26:06,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:26:06,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:26:06,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39139.72 MB 2025-02-15 05:26:06,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43271.11 MB 2025-02-15 05:26:06,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:26:06,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45585.79 MB 2025-02-15 05:26:06,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51248.10 MB 2025-02-15 05:26:06,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:26:06,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48815.39 MB 2025-02-15 05:26:06,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:26:06,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:26:06,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:26:06,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44804.65 MB 2025-02-15 05:26:06,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45571.65 MB 2025-02-15 05:26:06,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:26:06,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51248.10 MB 2025-02-15 05:26:06,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51667.53 MB 2025-02-15 05:26:06,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:26:06,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46279.44 MB 2025-02-15 05:26:06,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:26:06,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:26:06,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:26:06,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45984.54 MB 2025-02-15 05:26:06,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46212.77 MB 2025-02-15 05:26:06,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 05:26:06,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51667.53 MB 2025-02-15 05:26:06,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51667.53 MB 2025-02-15 05:26:06,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:26:06,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46427.28 MB 2025-02-15 05:26:06,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:26:06,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:26:06,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.49 seconds 2025-02-15 05:26:06,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33918.81 MB 2025-02-15 05:26:06,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46412.90 MB 2025-02-15 05:26:06,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12494.10 MB 2025-02-15 05:26:06,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66521.66 MB 2025-02-15 05:26:06,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51667.53 MB 2025-02-15 05:26:06,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14854.13 MB 2025-02-15 05:26:06,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46427.28 MB 2025-02-15 05:26:06,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:26:06,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:26:06,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:26:06,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46412.90 MB 2025-02-15 05:26:06,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38908.72 MB 2025-02-15 05:26:06,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7504.18 MB 2025-02-15 05:26:06,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51667.53 MB 2025-02-15 05:26:06,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51667.53 MB 2025-02-15 05:26:06,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:26:06,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48912.90 MB 2025-02-15 05:26:06,716 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 05:26:06,716 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:26:06,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:26:06,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:26:06,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:26:06,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:26:06,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38908.72 MB 2025-02-15 05:26:06,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47309.58 MB 2025-02-15 05:26:06,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 05:26:06,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51667.53 MB 2025-02-15 05:26:06,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60018.39 MB 2025-02-15 05:26:06,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 05:26:06,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47309.58 MB 2025-02-15 05:26:06,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 05:26:06,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:26:06,885 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:26:06,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:26:06,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:26:06,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:26:06,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:26:06,892 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:26:06,892 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:27:07,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:07,617 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:27:07,622 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:27:07,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:07,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1288, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:27:07,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:07,627 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1288, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:27:27,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:27:27,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:27:27,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.82 seconds 2025-02-15 05:27:27,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:27,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39071.76 MB 2025-02-15 05:27:27,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43630.97 MB 2025-02-15 05:27:27,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4559.21 MB 2025-02-15 05:27:27,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68369.25 MB 2025-02-15 05:27:27,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52900.66 MB 2025-02-15 05:27:27,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15468.59 MB 2025-02-15 05:27:27,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52620.00 MB 2025-02-15 05:27:27,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:27:27,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:27:27,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:27:27,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:27,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43630.97 MB 2025-02-15 05:27:27,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39601.83 MB 2025-02-15 05:27:27,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4029.14 MB 2025-02-15 05:27:27,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52900.66 MB 2025-02-15 05:27:27,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61949.87 MB 2025-02-15 05:27:27,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9049.21 MB 2025-02-15 05:27:27,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57237.57 MB 2025-02-15 05:27:29,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:27:29,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:27:29,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:27:29,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39601.83 MB 2025-02-15 05:27:29,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40132.67 MB 2025-02-15 05:27:29,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:27:29,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61949.87 MB 2025-02-15 05:27:29,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:27:29,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13608.42 MB 2025-02-15 05:27:29,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44111.22 MB 2025-02-15 05:27:29,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:27:29,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:27:29,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:27:29,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40132.67 MB 2025-02-15 05:27:29,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42022.20 MB 2025-02-15 05:27:29,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:27:29,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:27:29,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 05:27:29,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:27:29,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43439.63 MB 2025-02-15 05:27:29,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:27:29,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:27:29,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:27:29,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42022.20 MB 2025-02-15 05:27:29,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44264.06 MB 2025-02-15 05:27:29,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:27:29,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:27:29,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52116.32 MB 2025-02-15 05:27:29,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:27:29,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49808.34 MB 2025-02-15 05:27:29,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:27:29,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:27:29,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:27:29,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40132.67 MB 2025-02-15 05:27:29,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44264.06 MB 2025-02-15 05:27:29,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:27:29,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 05:27:29,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52116.32 MB 2025-02-15 05:27:29,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:27:29,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49808.34 MB 2025-02-15 05:27:29,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:27:29,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:27:29,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:27:29,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45797.60 MB 2025-02-15 05:27:29,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46564.60 MB 2025-02-15 05:27:29,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:27:29,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52116.32 MB 2025-02-15 05:27:29,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52535.75 MB 2025-02-15 05:27:29,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:27:29,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47272.39 MB 2025-02-15 05:27:29,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:27:29,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:27:29,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:27:29,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46977.49 MB 2025-02-15 05:27:29,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47205.52 MB 2025-02-15 05:27:29,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-15 05:27:29,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52535.75 MB 2025-02-15 05:27:29,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52535.75 MB 2025-02-15 05:27:29,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:27:29,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47445.42 MB 2025-02-15 05:27:29,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:27:29,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:27:29,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.23 seconds 2025-02-15 05:27:29,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:29,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34584.27 MB 2025-02-15 05:27:29,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47405.46 MB 2025-02-15 05:27:29,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12821.19 MB 2025-02-15 05:27:29,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68369.25 MB 2025-02-15 05:27:29,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52535.75 MB 2025-02-15 05:27:29,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15833.50 MB 2025-02-15 05:27:29,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47445.42 MB 2025-02-15 05:27:30,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:27:30,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:27:30,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:27:30,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:30,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47405.46 MB 2025-02-15 05:27:30,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39571.13 MB 2025-02-15 05:27:30,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7834.33 MB 2025-02-15 05:27:30,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52535.75 MB 2025-02-15 05:27:30,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52535.75 MB 2025-02-15 05:27:30,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:27:30,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49903.00 MB 2025-02-15 05:27:30,145 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 05:27:30,146 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:27:30,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:27:30,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:27:30,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:27:30,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:27:30,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39571.13 MB 2025-02-15 05:27:30,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47962.69 MB 2025-02-15 05:27:30,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.56 MB 2025-02-15 05:27:30,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52535.75 MB 2025-02-15 05:27:30,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56706.99 MB 2025-02-15 05:27:30,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 05:27:30,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47962.69 MB 2025-02-15 05:27:30,308 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 05:27:30,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:30,309 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:27:30,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:30,310 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:27:30,315 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:27:30,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:27:30,316 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:27:30,316 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:28:38,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:28:38,264 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:28:38,269 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:28:38,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:28:38,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1470, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:28:38,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:28:38,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1470, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:29:00,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:29:00,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:29:00,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.63 seconds 2025-02-15 05:29:00,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:00,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40339.97 MB 2025-02-15 05:29:00,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45543.00 MB 2025-02-15 05:29:00,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5203.03 MB 2025-02-15 05:29:00,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65049.46 MB 2025-02-15 05:29:00,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53527.71 MB 2025-02-15 05:29:00,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11521.75 MB 2025-02-15 05:29:00,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54341.19 MB 2025-02-15 05:29:01,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:29:01,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:29:01,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:29:01,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:01,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45543.00 MB 2025-02-15 05:29:01,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40547.99 MB 2025-02-15 05:29:01,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4995.01 MB 2025-02-15 05:29:01,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53527.71 MB 2025-02-15 05:29:01,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63587.75 MB 2025-02-15 05:29:01,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10060.04 MB 2025-02-15 05:29:01,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60670.82 MB 2025-02-15 05:29:02,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:29:02,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:29:02,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:29:02,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:02,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40547.99 MB 2025-02-15 05:29:02,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41078.83 MB 2025-02-15 05:29:02,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:29:02,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63587.75 MB 2025-02-15 05:29:02,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44153.44 MB 2025-02-15 05:29:02,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19434.31 MB 2025-02-15 05:29:02,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45057.38 MB 2025-02-15 05:29:02,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:29:02,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:29:02,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:29:02,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:02,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41078.83 MB 2025-02-15 05:29:02,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42968.36 MB 2025-02-15 05:29:02,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:29:02,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44153.44 MB 2025-02-15 05:29:02,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46984.59 MB 2025-02-15 05:29:02,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:29:02,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44385.79 MB 2025-02-15 05:29:03,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:29:03,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:29:03,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:29:03,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42968.36 MB 2025-02-15 05:29:03,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45210.22 MB 2025-02-15 05:29:03,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:29:03,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46984.59 MB 2025-02-15 05:29:03,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53118.76 MB 2025-02-15 05:29:03,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:29:03,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50754.50 MB 2025-02-15 05:29:03,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:29:03,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:29:03,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:29:03,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41078.83 MB 2025-02-15 05:29:03,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45210.22 MB 2025-02-15 05:29:03,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:29:03,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44153.44 MB 2025-02-15 05:29:03,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53118.76 MB 2025-02-15 05:29:03,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 05:29:03,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50754.50 MB 2025-02-15 05:29:03,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:29:03,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:29:03,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:29:03,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46743.76 MB 2025-02-15 05:29:03,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47510.76 MB 2025-02-15 05:29:03,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:29:03,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53118.76 MB 2025-02-15 05:29:03,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53538.19 MB 2025-02-15 05:29:03,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:29:03,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48218.55 MB 2025-02-15 05:29:03,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:29:03,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:29:03,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:29:03,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47923.65 MB 2025-02-15 05:29:03,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48151.74 MB 2025-02-15 05:29:03,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.09 MB 2025-02-15 05:29:03,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53538.19 MB 2025-02-15 05:29:03,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53538.19 MB 2025-02-15 05:29:03,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:03,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48387.31 MB 2025-02-15 05:29:03,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:29:03,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:29:03,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.08 seconds 2025-02-15 05:29:03,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35218.37 MB 2025-02-15 05:29:03,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48351.95 MB 2025-02-15 05:29:03,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13133.58 MB 2025-02-15 05:29:03,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65049.46 MB 2025-02-15 05:29:03,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53538.19 MB 2025-02-15 05:29:03,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11511.27 MB 2025-02-15 05:29:03,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48387.31 MB 2025-02-15 05:29:03,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:29:03,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:29:03,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:29:03,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48351.95 MB 2025-02-15 05:29:03,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40209.43 MB 2025-02-15 05:29:03,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8142.53 MB 2025-02-15 05:29:03,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53538.19 MB 2025-02-15 05:29:03,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53538.19 MB 2025-02-15 05:29:03,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:03,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50852.87 MB 2025-02-15 05:29:03,639 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-15 05:29:03,639 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:29:03,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:29:03,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:29:03,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:29:03,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:03,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40209.43 MB 2025-02-15 05:29:03,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48612.99 MB 2025-02-15 05:29:03,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-15 05:29:03,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53538.19 MB 2025-02-15 05:29:03,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61893.25 MB 2025-02-15 05:29:03,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 05:29:03,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48612.99 MB 2025-02-15 05:29:03,803 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-15 05:29:03,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:03,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:29:03,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:03,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:29:03,811 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:29:03,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:03,812 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:29:03,812 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-15 05:29:13,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:13,668 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:29:13,672 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:29:13,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:13,676 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:29:13,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:13,677 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:29:35,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:29:35,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:29:35,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.61 seconds 2025-02-15 05:29:35,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:35,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39754.64 MB 2025-02-15 05:29:35,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44659.88 MB 2025-02-15 05:29:35,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4905.24 MB 2025-02-15 05:29:35,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70248.30 MB 2025-02-15 05:29:35,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53236.20 MB 2025-02-15 05:29:35,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17012.10 MB 2025-02-15 05:29:35,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53529.37 MB 2025-02-15 05:29:35,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:29:35,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:29:35,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:29:35,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:35,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44659.88 MB 2025-02-15 05:29:35,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40111.30 MB 2025-02-15 05:29:35,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4548.58 MB 2025-02-15 05:29:35,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53236.20 MB 2025-02-15 05:29:35,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62365.11 MB 2025-02-15 05:29:35,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9128.90 MB 2025-02-15 05:29:35,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58380.87 MB 2025-02-15 05:29:37,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:29:37,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:29:37,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:29:37,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40111.30 MB 2025-02-15 05:29:37,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40642.14 MB 2025-02-15 05:29:37,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:29:37,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62365.11 MB 2025-02-15 05:29:37,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48330.96 MB 2025-02-15 05:29:37,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14034.14 MB 2025-02-15 05:29:37,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44620.69 MB 2025-02-15 05:29:37,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:29:37,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:29:37,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:29:37,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40642.14 MB 2025-02-15 05:29:37,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42531.67 MB 2025-02-15 05:29:37,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:29:37,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48330.96 MB 2025-02-15 05:29:37,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48330.96 MB 2025-02-15 05:29:37,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:37,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43949.10 MB 2025-02-15 05:29:37,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:29:37,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:29:37,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:29:37,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42531.67 MB 2025-02-15 05:29:37,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44773.53 MB 2025-02-15 05:29:37,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:29:37,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48330.96 MB 2025-02-15 05:29:37,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53049.56 MB 2025-02-15 05:29:37,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:29:37,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50317.81 MB 2025-02-15 05:29:37,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:29:37,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:29:37,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:29:37,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40642.14 MB 2025-02-15 05:29:37,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44773.53 MB 2025-02-15 05:29:37,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:29:37,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48330.96 MB 2025-02-15 05:29:37,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53049.56 MB 2025-02-15 05:29:37,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:29:37,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50317.81 MB 2025-02-15 05:29:37,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:29:37,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:29:37,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:29:37,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46307.07 MB 2025-02-15 05:29:37,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47074.07 MB 2025-02-15 05:29:37,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:29:37,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53049.56 MB 2025-02-15 05:29:37,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53468.99 MB 2025-02-15 05:29:37,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 05:29:37,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47781.86 MB 2025-02-15 05:29:37,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:29:37,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:29:37,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:29:37,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47486.96 MB 2025-02-15 05:29:37,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47715.09 MB 2025-02-15 05:29:37,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-15 05:29:37,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53468.99 MB 2025-02-15 05:29:37,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53468.99 MB 2025-02-15 05:29:37,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:37,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47931.68 MB 2025-02-15 05:29:37,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:29:37,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:29:37,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.15 seconds 2025-02-15 05:29:37,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:37,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34925.71 MB 2025-02-15 05:29:37,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47915.13 MB 2025-02-15 05:29:37,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.42 MB 2025-02-15 05:29:37,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70248.30 MB 2025-02-15 05:29:37,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53468.99 MB 2025-02-15 05:29:37,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16779.31 MB 2025-02-15 05:29:37,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47931.68 MB 2025-02-15 05:29:38,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:29:38,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:29:38,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 05:29:38,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:38,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47915.13 MB 2025-02-15 05:29:38,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39914.10 MB 2025-02-15 05:29:38,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8001.03 MB 2025-02-15 05:29:38,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53468.99 MB 2025-02-15 05:29:38,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53468.99 MB 2025-02-15 05:29:38,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:38,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50413.89 MB 2025-02-15 05:29:38,142 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 05:29:38,143 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:29:38,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:29:38,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:29:38,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:29:38,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:38,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39914.10 MB 2025-02-15 05:29:38,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48310.74 MB 2025-02-15 05:29:38,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-15 05:29:38,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53468.99 MB 2025-02-15 05:29:38,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61815.65 MB 2025-02-15 05:29:38,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 05:29:38,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48310.74 MB 2025-02-15 05:29:38,321 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 05:29:38,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:38,323 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:29:38,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:38,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:29:38,328 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:29:38,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:38,329 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:29:38,329 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 05:29:49,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:49,229 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:29:49,234 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:29:49,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:49,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 134, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:29:49,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:49,238 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 134, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:29:51,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:29:51,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:29:51,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.13 seconds 2025-02-15 05:29:51,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:51,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31030.51 MB 2025-02-15 05:29:51,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31504.72 MB 2025-02-15 05:29:51,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 474.22 MB 2025-02-15 05:29:51,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70162.32 MB 2025-02-15 05:29:51,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:51,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32935.77 MB 2025-02-15 05:29:51,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40501.88 MB 2025-02-15 05:29:51,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:29:51,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:29:51,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:29:51,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:51,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31504.72 MB 2025-02-15 05:29:51,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31734.48 MB 2025-02-15 05:29:51,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.76 MB 2025-02-15 05:29:51,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:51,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:51,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:51,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33386.96 MB 2025-02-15 05:29:52,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:29:52,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:29:52,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-15 05:29:52,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31734.48 MB 2025-02-15 05:29:52,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31912.31 MB 2025-02-15 05:29:52,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-15 05:29:52,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:52,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:52,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35904.13 MB 2025-02-15 05:29:52,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:29:52,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:29:52,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:29:52,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31912.25 MB 2025-02-15 05:29:52,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32545.09 MB 2025-02-15 05:29:52,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-15 05:29:52,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:52,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:52,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33019.93 MB 2025-02-15 05:29:52,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:29:52,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:29:52,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:29:52,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32545.09 MB 2025-02-15 05:29:52,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33296.15 MB 2025-02-15 05:29:52,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-15 05:29:52,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:52,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:52,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35153.45 MB 2025-02-15 05:29:52,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:29:52,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:29:52,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:29:52,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31912.25 MB 2025-02-15 05:29:52,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33296.15 MB 2025-02-15 05:29:52,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-15 05:29:52,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:52,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-15 05:29:52,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35153.45 MB 2025-02-15 05:29:52,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:29:52,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:29:52,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:29:52,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33809.89 MB 2025-02-15 05:29:52,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34066.84 MB 2025-02-15 05:29:52,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-15 05:29:52,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-15 05:29:52,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37367.05 MB 2025-02-15 05:29:52,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 140.51 MB 2025-02-15 05:29:52,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34315.46 MB 2025-02-15 05:29:52,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:29:52,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:29:52,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:29:52,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34205.16 MB 2025-02-15 05:29:52,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34426.12 MB 2025-02-15 05:29:52,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.95 MB 2025-02-15 05:29:52,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37367.05 MB 2025-02-15 05:29:52,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37367.05 MB 2025-02-15 05:29:52,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34429.14 MB 2025-02-15 05:29:52,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:29:52,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:29:52,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.95 seconds 2025-02-15 05:29:52,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30563.64 MB 2025-02-15 05:29:52,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31298.49 MB 2025-02-15 05:29:52,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.85 MB 2025-02-15 05:29:52,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70162.32 MB 2025-02-15 05:29:52,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37367.05 MB 2025-02-15 05:29:52,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32795.26 MB 2025-02-15 05:29:52,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34626.84 MB 2025-02-15 05:29:52,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:29:52,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:29:52,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:29:52,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31298.49 MB 2025-02-15 05:29:52,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34307.36 MB 2025-02-15 05:29:52,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.87 MB 2025-02-15 05:29:52,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37367.05 MB 2025-02-15 05:29:52,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37367.05 MB 2025-02-15 05:29:52,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:29:52,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34608.21 MB 2025-02-15 05:29:52,479 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 05:29:52,479 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 05:29:52,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:29:52,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:29:52,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:29:52,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:29:52,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34307.36 MB 2025-02-15 05:29:52,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42732.31 MB 2025-02-15 05:29:52,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 05:29:52,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37367.05 MB 2025-02-15 05:29:52,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47838.13 MB 2025-02-15 05:29:52,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 05:29:52,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42732.31 MB 2025-02-15 05:29:52,644 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 05:29:52,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:52,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:29:52,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:52,647 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:29:52,651 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:29:52,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:29:52,653 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:29:52,653 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 05:30:29,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:29,987 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:30:29,992 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:30:29,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:29,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 206, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:30:29,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:29,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 206, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:30:33,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:30:33,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:30:33,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.19 seconds 2025-02-15 05:30:33,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:33,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36869.57 MB 2025-02-15 05:30:33,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37598.60 MB 2025-02-15 05:30:33,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 729.02 MB 2025-02-15 05:30:33,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56214.16 MB 2025-02-15 05:30:33,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41414.56 MB 2025-02-15 05:30:33,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14799.60 MB 2025-02-15 05:30:33,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46567.44 MB 2025-02-15 05:30:33,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:30:33,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:30:33,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:30:33,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:33,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37598.60 MB 2025-02-15 05:30:33,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37952.53 MB 2025-02-15 05:30:33,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.93 MB 2025-02-15 05:30:33,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41414.56 MB 2025-02-15 05:30:33,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43228.59 MB 2025-02-15 05:30:33,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1814.04 MB 2025-02-15 05:30:33,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40493.65 MB 2025-02-15 05:30:34,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:30:34,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:30:34,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.99 seconds 2025-02-15 05:30:34,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37952.53 MB 2025-02-15 05:30:34,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38225.91 MB 2025-02-15 05:30:34,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.38 MB 2025-02-15 05:30:34,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43228.59 MB 2025-02-15 05:30:34,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42901.44 MB 2025-02-15 05:30:34,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -327.16 MB 2025-02-15 05:30:34,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42208.15 MB 2025-02-15 05:30:34,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:30:34,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:30:34,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:30:34,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38225.91 MB 2025-02-15 05:30:34,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39198.78 MB 2025-02-15 05:30:34,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 972.87 MB 2025-02-15 05:30:34,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42901.44 MB 2025-02-15 05:30:34,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42901.44 MB 2025-02-15 05:30:34,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:30:34,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39928.76 MB 2025-02-15 05:30:34,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:30:34,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:30:34,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:30:34,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39198.78 MB 2025-02-15 05:30:34,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40353.37 MB 2025-02-15 05:30:34,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1154.59 MB 2025-02-15 05:30:34,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42901.44 MB 2025-02-15 05:30:34,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45820.67 MB 2025-02-15 05:30:34,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2919.24 MB 2025-02-15 05:30:34,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43211.27 MB 2025-02-15 05:30:34,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:30:34,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:30:34,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:30:34,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38225.91 MB 2025-02-15 05:30:34,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40353.37 MB 2025-02-15 05:30:34,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2127.46 MB 2025-02-15 05:30:34,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42901.44 MB 2025-02-15 05:30:34,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45820.67 MB 2025-02-15 05:30:34,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2919.24 MB 2025-02-15 05:30:34,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43211.27 MB 2025-02-15 05:30:34,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:30:34,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:30:34,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:30:34,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41143.14 MB 2025-02-15 05:30:34,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41538.67 MB 2025-02-15 05:30:34,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 395.53 MB 2025-02-15 05:30:34,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45820.67 MB 2025-02-15 05:30:34,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46036.68 MB 2025-02-15 05:30:34,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-15 05:30:34,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41904.70 MB 2025-02-15 05:30:34,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:30:34,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:30:34,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:30:34,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41751.32 MB 2025-02-15 05:30:34,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41968.66 MB 2025-02-15 05:30:34,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.34 MB 2025-02-15 05:30:34,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46036.68 MB 2025-02-15 05:30:34,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46036.68 MB 2025-02-15 05:30:34,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:30:34,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42008.17 MB 2025-02-15 05:30:34,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:30:34,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:30:34,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-15 05:30:34,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36151.85 MB 2025-02-15 05:30:34,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42169.73 MB 2025-02-15 05:30:34,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6017.88 MB 2025-02-15 05:30:34,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56214.16 MB 2025-02-15 05:30:34,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46036.68 MB 2025-02-15 05:30:34,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10177.48 MB 2025-02-15 05:30:34,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42169.73 MB 2025-02-15 05:30:34,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:30:34,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:30:34,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:30:34,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37227.19 MB 2025-02-15 05:30:34,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40242.01 MB 2025-02-15 05:30:34,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.82 MB 2025-02-15 05:30:34,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46036.68 MB 2025-02-15 05:30:34,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46036.68 MB 2025-02-15 05:30:34,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:30:34,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40543.38 MB 2025-02-15 05:30:34,697 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:30:34,697 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:30:34,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:30:34,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:30:34,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:30:34,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:30:34,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40242.01 MB 2025-02-15 05:30:34,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48681.03 MB 2025-02-15 05:30:34,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:30:34,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46036.68 MB 2025-02-15 05:30:34,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56526.64 MB 2025-02-15 05:30:34,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:30:34,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48681.03 MB 2025-02-15 05:30:34,876 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:30:34,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:34,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:30:34,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:34,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:30:34,883 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:30:34,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:30:34,884 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:30:34,884 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:31:25,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:25,353 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:31:25,360 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:31:25,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:25,367 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 865, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:31:25,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:25,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 865, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:31:38,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:31:38,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:31:38,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.37 seconds 2025-02-15 05:31:38,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:38,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41461.59 MB 2025-02-15 05:31:38,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44523.43 MB 2025-02-15 05:31:38,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3061.84 MB 2025-02-15 05:31:38,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69111.64 MB 2025-02-15 05:31:38,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50769.95 MB 2025-02-15 05:31:38,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18341.69 MB 2025-02-15 05:31:38,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53425.18 MB 2025-02-15 05:31:38,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:31:38,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:31:38,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:31:38,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:38,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44523.43 MB 2025-02-15 05:31:38,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42740.14 MB 2025-02-15 05:31:38,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1783.29 MB 2025-02-15 05:31:38,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50769.95 MB 2025-02-15 05:31:38,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57963.18 MB 2025-02-15 05:31:38,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7193.23 MB 2025-02-15 05:31:38,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54522.06 MB 2025-02-15 05:31:40,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:31:40,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:31:40,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 05:31:40,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:40,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42740.14 MB 2025-02-15 05:31:40,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43270.98 MB 2025-02-15 05:31:40,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:31:40,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57963.18 MB 2025-02-15 05:31:40,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49123.69 MB 2025-02-15 05:31:40,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8839.50 MB 2025-02-15 05:31:40,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47249.53 MB 2025-02-15 05:31:40,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:31:40,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:31:40,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:31:40,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:40,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43270.98 MB 2025-02-15 05:31:40,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45160.52 MB 2025-02-15 05:31:40,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:31:40,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49123.69 MB 2025-02-15 05:31:40,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50067.41 MB 2025-02-15 05:31:40,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 05:31:40,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46577.95 MB 2025-02-15 05:31:40,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:31:40,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:31:40,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:31:40,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:40,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45160.52 MB 2025-02-15 05:31:40,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47402.37 MB 2025-02-15 05:31:40,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:31:40,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50067.41 MB 2025-02-15 05:31:40,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55729.72 MB 2025-02-15 05:31:40,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:31:40,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52947.58 MB 2025-02-15 05:31:40,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:31:40,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:31:40,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:31:40,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:40,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43270.98 MB 2025-02-15 05:31:40,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47402.37 MB 2025-02-15 05:31:40,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:31:40,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49123.69 MB 2025-02-15 05:31:40,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55729.72 MB 2025-02-15 05:31:40,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 05:31:40,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52947.58 MB 2025-02-15 05:31:41,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:31:41,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:31:41,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.32 seconds 2025-02-15 05:31:41,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:41,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48935.92 MB 2025-02-15 05:31:41,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27238.42 MB 2025-02-15 05:31:41,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -21697.50 MB 2025-02-15 05:31:41,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55729.72 MB 2025-02-15 05:31:41,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55828.28 MB 2025-02-15 05:31:41,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 98.57 MB 2025-02-15 05:31:41,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49137.78 MB 2025-02-15 05:31:41,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:31:41,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:31:41,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:31:41,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:41,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27651.31 MB 2025-02-15 05:31:41,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27878.82 MB 2025-02-15 05:31:41,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.51 MB 2025-02-15 05:31:41,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55828.28 MB 2025-02-15 05:31:41,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55828.28 MB 2025-02-15 05:31:41,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:31:41,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28098.17 MB 2025-02-15 05:31:41,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:31:41,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:31:41,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.91 seconds 2025-02-15 05:31:41,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:41,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38447.86 MB 2025-02-15 05:31:41,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28079.42 MB 2025-02-15 05:31:41,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10368.44 MB 2025-02-15 05:31:41,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69111.64 MB 2025-02-15 05:31:41,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55828.28 MB 2025-02-15 05:31:41,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13283.36 MB 2025-02-15 05:31:41,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28098.17 MB 2025-02-15 05:31:41,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:31:41,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:31:41,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:31:41,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:41,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28079.42 MB 2025-02-15 05:31:41,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20980.93 MB 2025-02-15 05:31:41,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7098.49 MB 2025-02-15 05:31:41,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55828.28 MB 2025-02-15 05:31:41,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55828.28 MB 2025-02-15 05:31:41,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:31:41,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28079.42 MB 2025-02-15 05:31:41,570 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 05:31:41,570 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:31:41,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:31:41,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:31:41,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:31:41,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:31:41,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20980.93 MB 2025-02-15 05:31:41,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29400.01 MB 2025-02-15 05:31:41,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 05:31:41,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55828.28 MB 2025-02-15 05:31:41,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55828.28 MB 2025-02-15 05:31:41,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:31:41,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29400.01 MB 2025-02-15 05:31:41,734 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 05:31:41,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:41,736 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:31:41,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:41,736 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:31:41,741 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:31:41,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:41,742 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:31:41,742 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:31:56,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:56,007 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:31:56,012 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:31:56,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:56,015 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1297, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:31:56,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:31:56,016 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1297, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:32:16,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:32:16,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:32:16,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.05 seconds 2025-02-15 05:32:16,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:16,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30594.33 MB 2025-02-15 05:32:16,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35185.00 MB 2025-02-15 05:32:16,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4590.67 MB 2025-02-15 05:32:16,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60014.20 MB 2025-02-15 05:32:16,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48775.56 MB 2025-02-15 05:32:16,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11238.64 MB 2025-02-15 05:32:16,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44142.57 MB 2025-02-15 05:32:16,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:32:16,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:32:16,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:32:16,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:16,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35185.00 MB 2025-02-15 05:32:16,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22520.55 MB 2025-02-15 05:32:16,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12664.45 MB 2025-02-15 05:32:16,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48775.56 MB 2025-02-15 05:32:16,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42010.15 MB 2025-02-15 05:32:16,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6765.41 MB 2025-02-15 05:32:16,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38423.51 MB 2025-02-15 05:32:18,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:32:18,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:32:18,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:32:18,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22520.55 MB 2025-02-15 05:32:18,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23051.39 MB 2025-02-15 05:32:18,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:32:18,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42010.15 MB 2025-02-15 05:32:18,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 05:32:18,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12954.11 MB 2025-02-15 05:32:18,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27029.94 MB 2025-02-15 05:32:18,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:32:18,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:32:18,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:32:18,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23051.39 MB 2025-02-15 05:32:18,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24940.92 MB 2025-02-15 05:32:18,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:32:18,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 05:32:18,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 05:32:18,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:32:18,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26358.35 MB 2025-02-15 05:32:18,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:32:18,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:32:18,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:32:18,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24940.92 MB 2025-02-15 05:32:18,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27182.78 MB 2025-02-15 05:32:18,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:32:18,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 05:32:18,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 05:32:18,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:32:18,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32727.06 MB 2025-02-15 05:32:18,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:32:18,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:32:18,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:32:18,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23051.39 MB 2025-02-15 05:32:18,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27182.78 MB 2025-02-15 05:32:18,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:32:18,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 05:32:18,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34718.35 MB 2025-02-15 05:32:18,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:32:18,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32727.06 MB 2025-02-15 05:32:18,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:32:18,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:32:18,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:32:18,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28716.32 MB 2025-02-15 05:32:18,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29483.32 MB 2025-02-15 05:32:18,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:32:18,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34718.35 MB 2025-02-15 05:32:18,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 05:32:18,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 05:32:18,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30191.11 MB 2025-02-15 05:32:18,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:32:18,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:32:18,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:32:18,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29896.21 MB 2025-02-15 05:32:18,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30125.34 MB 2025-02-15 05:32:18,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.12 MB 2025-02-15 05:32:18,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 05:32:18,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 05:32:18,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:32:18,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30346.98 MB 2025-02-15 05:32:18,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:32:18,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:32:18,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.46 seconds 2025-02-15 05:32:18,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26075.48 MB 2025-02-15 05:32:18,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30326.41 MB 2025-02-15 05:32:18,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-15 05:32:18,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60014.20 MB 2025-02-15 05:32:18,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 05:32:18,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24884.81 MB 2025-02-15 05:32:18,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30346.98 MB 2025-02-15 05:32:18,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:32:18,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:32:18,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:32:18,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30326.41 MB 2025-02-15 05:32:18,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22491.95 MB 2025-02-15 05:32:18,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7834.46 MB 2025-02-15 05:32:18,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 05:32:18,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 05:32:18,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:32:18,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32034.34 MB 2025-02-15 05:32:18,765 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:32:18,766 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:32:18,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:32:18,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:32:18,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:32:18,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:18,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22491.95 MB 2025-02-15 05:32:18,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30930.97 MB 2025-02-15 05:32:18,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:32:18,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 05:32:18,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43520.10 MB 2025-02-15 05:32:18,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:32:18,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30930.97 MB 2025-02-15 05:32:18,930 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:32:18,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:18,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:32:18,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:18,932 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:32:18,937 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:32:18,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:18,938 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:32:18,938 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:32:33,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:33,536 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:32:33,541 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:32:33,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:33,545 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 287, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:32:33,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:33,546 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 287, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:32:38,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:32:38,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:32:38,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.48 seconds 2025-02-15 05:32:38,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:38,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14968.57 MB 2025-02-15 05:32:38,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15984.25 MB 2025-02-15 05:32:38,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1015.68 MB 2025-02-15 05:32:38,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56105.11 MB 2025-02-15 05:32:38,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-15 05:32:38,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35154.56 MB 2025-02-15 05:32:38,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24892.92 MB 2025-02-15 05:32:38,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:32:38,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:32:38,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:32:38,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:38,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15984.25 MB 2025-02-15 05:32:38,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16329.05 MB 2025-02-15 05:32:38,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.81 MB 2025-02-15 05:32:38,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-15 05:32:38,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22821.21 MB 2025-02-15 05:32:38,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1870.66 MB 2025-02-15 05:32:38,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19720.76 MB 2025-02-15 05:32:39,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:32:39,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:32:39,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.29 seconds 2025-02-15 05:32:39,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16329.05 MB 2025-02-15 05:32:39,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16682.06 MB 2025-02-15 05:32:39,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.01 MB 2025-02-15 05:32:39,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22821.21 MB 2025-02-15 05:32:39,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-15 05:32:39,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2581.59 MB 2025-02-15 05:32:39,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20669.61 MB 2025-02-15 05:32:39,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:32:39,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:32:39,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:32:39,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16682.06 MB 2025-02-15 05:32:39,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.30 MB 2025-02-15 05:32:39,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.24 MB 2025-02-15 05:32:39,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-15 05:32:39,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20868.76 MB 2025-02-15 05:32:39,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 629.15 MB 2025-02-15 05:32:39,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18880.89 MB 2025-02-15 05:32:39,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:32:39,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:32:39,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 05:32:39,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.30 MB 2025-02-15 05:32:39,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.16 MB 2025-02-15 05:32:39,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1490.86 MB 2025-02-15 05:32:39,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20868.76 MB 2025-02-15 05:32:39,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24643.63 MB 2025-02-15 05:32:39,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:32:39,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23117.66 MB 2025-02-15 05:32:39,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:32:39,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:32:39,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:32:39,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16682.06 MB 2025-02-15 05:32:39,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.16 MB 2025-02-15 05:32:39,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2747.10 MB 2025-02-15 05:32:39,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-15 05:32:39,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24643.63 MB 2025-02-15 05:32:39,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4404.02 MB 2025-02-15 05:32:39,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23117.66 MB 2025-02-15 05:32:39,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:32:39,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:32:39,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:32:39,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20449.75 MB 2025-02-15 05:32:39,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20960.59 MB 2025-02-15 05:32:39,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 510.84 MB 2025-02-15 05:32:39,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24643.63 MB 2025-02-15 05:32:39,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24916.26 MB 2025-02-15 05:32:39,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 272.63 MB 2025-02-15 05:32:39,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21431.27 MB 2025-02-15 05:32:39,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:32:39,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:32:39,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:32:39,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21235.17 MB 2025-02-15 05:32:39,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21439.89 MB 2025-02-15 05:32:39,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.72 MB 2025-02-15 05:32:39,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24916.26 MB 2025-02-15 05:32:39,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24920.46 MB 2025-02-15 05:32:39,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 05:32:39,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21501.72 MB 2025-02-15 05:32:39,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:32:39,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:32:39,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.10 seconds 2025-02-15 05:32:39,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13968.64 MB 2025-02-15 05:32:39,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21640.96 MB 2025-02-15 05:32:39,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7672.32 MB 2025-02-15 05:32:39,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56105.11 MB 2025-02-15 05:32:39,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24920.46 MB 2025-02-15 05:32:39,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31184.65 MB 2025-02-15 05:32:39,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21640.96 MB 2025-02-15 05:32:39,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:32:39,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:32:39,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:32:39,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.96 MB 2025-02-15 05:32:39,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24654.99 MB 2025-02-15 05:32:39,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 05:32:39,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24920.46 MB 2025-02-15 05:32:39,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25994.20 MB 2025-02-15 05:32:39,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-15 05:32:39,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24956.62 MB 2025-02-15 05:32:39,937 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:32:39,937 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:32:39,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:32:39,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:32:39,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:32:39,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:32:39,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18341.69 MB 2025-02-15 05:32:39,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26780.71 MB 2025-02-15 05:32:39,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:32:39,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25994.20 MB 2025-02-15 05:32:39,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36484.15 MB 2025-02-15 05:32:39,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:32:39,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26780.71 MB 2025-02-15 05:32:40,104 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:32:40,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:40,106 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:32:40,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:40,107 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:32:40,111 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:32:40,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:32:40,112 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:32:40,113 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:34:00,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:00,837 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:34:00,843 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:34:00,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:00,846 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:34:00,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:00,847 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:34:05,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:34:05,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:34:05,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.00 seconds 2025-02-15 05:34:05,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:05,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15226.39 MB 2025-02-15 05:34:05,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16373.01 MB 2025-02-15 05:34:05,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1146.62 MB 2025-02-15 05:34:05,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49069.16 MB 2025-02-15 05:34:05,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20631.78 MB 2025-02-15 05:34:05,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28437.38 MB 2025-02-15 05:34:05,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25378.21 MB 2025-02-15 05:34:05,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:34:05,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:34:05,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:34:05,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:05,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16373.01 MB 2025-02-15 05:34:05,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.25 MB 2025-02-15 05:34:05,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.24 MB 2025-02-15 05:34:05,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20631.78 MB 2025-02-15 05:34:05,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22842.18 MB 2025-02-15 05:34:05,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2210.40 MB 2025-02-15 05:34:05,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20757.47 MB 2025-02-15 05:34:07,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:34:07,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:34:07,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.50 seconds 2025-02-15 05:34:07,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16845.25 MB 2025-02-15 05:34:07,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.30 MB 2025-02-15 05:34:07,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 414.06 MB 2025-02-15 05:34:07,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22842.18 MB 2025-02-15 05:34:07,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19469.96 MB 2025-02-15 05:34:07,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3372.22 MB 2025-02-15 05:34:07,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21185.81 MB 2025-02-15 05:34:07,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:34:07,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:34:07,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:34:07,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17259.30 MB 2025-02-15 05:34:07,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18733.60 MB 2025-02-15 05:34:07,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1474.30 MB 2025-02-15 05:34:07,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19469.96 MB 2025-02-15 05:34:07,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21678.26 MB 2025-02-15 05:34:07,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2208.30 MB 2025-02-15 05:34:07,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19839.20 MB 2025-02-15 05:34:07,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:34:07,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:34:07,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:34:07,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18733.60 MB 2025-02-15 05:34:07,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20482.26 MB 2025-02-15 05:34:07,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1748.66 MB 2025-02-15 05:34:07,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21678.26 MB 2025-02-15 05:34:07,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26466.06 MB 2025-02-15 05:34:07,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4787.80 MB 2025-02-15 05:34:07,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.94 MB 2025-02-15 05:34:07,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:34:07,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:34:07,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:34:07,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17259.30 MB 2025-02-15 05:34:07,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20482.26 MB 2025-02-15 05:34:07,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3222.96 MB 2025-02-15 05:34:07,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19469.96 MB 2025-02-15 05:34:07,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26466.06 MB 2025-02-15 05:34:07,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6996.10 MB 2025-02-15 05:34:07,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.94 MB 2025-02-15 05:34:07,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:34:07,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:34:07,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:34:07,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21678.43 MB 2025-02-15 05:34:07,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22278.79 MB 2025-02-15 05:34:07,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 600.36 MB 2025-02-15 05:34:07,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26466.06 MB 2025-02-15 05:34:07,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-15 05:34:07,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 325.06 MB 2025-02-15 05:34:07,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22830.86 MB 2025-02-15 05:34:07,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:34:07,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:34:07,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:34:07,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22600.84 MB 2025-02-15 05:34:07,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22829.49 MB 2025-02-15 05:34:07,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-15 05:34:07,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26791.12 MB 2025-02-15 05:34:07,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-15 05:34:07,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:34:07,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22962.46 MB 2025-02-15 05:34:07,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:34:07,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:34:07,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.86 seconds 2025-02-15 05:34:07,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14097.55 MB 2025-02-15 05:34:07,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23030.56 MB 2025-02-15 05:34:07,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8933.01 MB 2025-02-15 05:34:07,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49069.16 MB 2025-02-15 05:34:07,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-15 05:34:07,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22278.05 MB 2025-02-15 05:34:07,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23030.56 MB 2025-02-15 05:34:07,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:34:07,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:34:07,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:34:07,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:07,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23030.56 MB 2025-02-15 05:34:07,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26044.59 MB 2025-02-15 05:34:07,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 05:34:07,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26791.12 MB 2025-02-15 05:34:07,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27462.21 MB 2025-02-15 05:34:07,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 671.09 MB 2025-02-15 05:34:07,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26346.22 MB 2025-02-15 05:34:07,993 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:34:07,994 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:34:08,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:34:08,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:34:08,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:34:08,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:34:08,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18688.74 MB 2025-02-15 05:34:08,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27127.76 MB 2025-02-15 05:34:08,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:34:08,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27462.21 MB 2025-02-15 05:34:08,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37952.16 MB 2025-02-15 05:34:08,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:34:08,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27127.76 MB 2025-02-15 05:34:08,158 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:34:08,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:08,160 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:34:08,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:08,161 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:34:08,165 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:34:08,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:34:08,166 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:34:08,166 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:35:31,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:35:31,109 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:35:31,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:35:31,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:35:31,123 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1860, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:35:31,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:35:31,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1860, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:35:59,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:35:59,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:35:59,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.70 seconds 2025-02-15 05:35:59,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:35:59,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.48 MB 2025-02-15 05:35:59,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32512.44 MB 2025-02-15 05:35:59,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6582.96 MB 2025-02-15 05:35:59,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50537.17 MB 2025-02-15 05:35:59,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39839.60 MB 2025-02-15 05:35:59,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10697.57 MB 2025-02-15 05:35:59,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41516.15 MB 2025-02-15 05:35:59,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:35:59,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:35:59,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 05:35:59,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:35:59,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32512.44 MB 2025-02-15 05:35:59,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25447.41 MB 2025-02-15 05:35:59,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7065.03 MB 2025-02-15 05:35:59,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39839.60 MB 2025-02-15 05:35:59,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53125.05 MB 2025-02-15 05:35:59,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13285.46 MB 2025-02-15 05:35:59,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50368.21 MB 2025-02-15 05:36:01,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:36:01,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:36:01,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:36:01,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:01,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25447.41 MB 2025-02-15 05:36:01,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25978.25 MB 2025-02-15 05:36:01,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:36:01,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53125.05 MB 2025-02-15 05:36:01,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 05:36:01,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18452.84 MB 2025-02-15 05:36:01,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29956.80 MB 2025-02-15 05:36:01,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:36:01,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:36:01,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:36:01,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:01,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25978.25 MB 2025-02-15 05:36:01,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27867.79 MB 2025-02-15 05:36:01,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:36:01,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 05:36:01,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 05:36:01,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:36:01,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29285.22 MB 2025-02-15 05:36:02,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:36:02,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:36:02,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:36:02,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27867.79 MB 2025-02-15 05:36:02,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30109.64 MB 2025-02-15 05:36:02,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:36:02,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 05:36:02,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38447.09 MB 2025-02-15 05:36:02,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:36:02,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35653.92 MB 2025-02-15 05:36:02,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:36:02,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:36:02,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:36:02,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25978.25 MB 2025-02-15 05:36:02,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30109.64 MB 2025-02-15 05:36:02,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:36:02,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 05:36:02,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38447.09 MB 2025-02-15 05:36:02,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:36:02,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35653.92 MB 2025-02-15 05:36:02,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:36:02,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:36:02,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:36:02,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31643.18 MB 2025-02-15 05:36:02,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32410.19 MB 2025-02-15 05:36:02,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:36:02,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38447.09 MB 2025-02-15 05:36:02,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38864.42 MB 2025-02-15 05:36:02,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:36:02,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33117.98 MB 2025-02-15 05:36:02,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:36:02,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:36:02,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:36:02,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32823.08 MB 2025-02-15 05:36:02,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33051.86 MB 2025-02-15 05:36:02,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 05:36:02,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38864.42 MB 2025-02-15 05:36:02,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38864.42 MB 2025-02-15 05:36:02,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:36:02,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33266.86 MB 2025-02-15 05:36:02,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:36:02,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:36:02,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.17 seconds 2025-02-15 05:36:02,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19449.10 MB 2025-02-15 05:36:02,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33252.57 MB 2025-02-15 05:36:02,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13803.47 MB 2025-02-15 05:36:02,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50537.17 MB 2025-02-15 05:36:02,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38864.42 MB 2025-02-15 05:36:02,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11672.75 MB 2025-02-15 05:36:02,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33266.86 MB 2025-02-15 05:36:02,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:36:02,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:36:02,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:36:02,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33252.57 MB 2025-02-15 05:36:02,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24447.92 MB 2025-02-15 05:36:02,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8804.65 MB 2025-02-15 05:36:02,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38864.42 MB 2025-02-15 05:36:02,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38864.42 MB 2025-02-15 05:36:02,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:36:02,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35759.77 MB 2025-02-15 05:36:02,589 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 05:36:02,589 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:36:02,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:36:02,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:36:02,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:36:02,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:36:02,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24447.92 MB 2025-02-15 05:36:02,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32871.13 MB 2025-02-15 05:36:02,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 05:36:02,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38864.42 MB 2025-02-15 05:36:02,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-15 05:36:02,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 05:36:02,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32871.13 MB 2025-02-15 05:36:02,752 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 05:36:02,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:36:02,753 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:36:02,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:36:02,754 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:36:02,759 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:36:02,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:36:02,760 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:36:02,760 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:37:19,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:19,005 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:37:19,010 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:37:19,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:19,015 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1880, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:37:19,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:19,016 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1880, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:37:48,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:37:48,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:37:48,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.98 seconds 2025-02-15 05:37:48,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:48,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26068.85 MB 2025-02-15 05:37:48,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32723.11 MB 2025-02-15 05:37:48,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6654.26 MB 2025-02-15 05:37:48,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51428.46 MB 2025-02-15 05:37:48,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39889.93 MB 2025-02-15 05:37:48,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11538.53 MB 2025-02-15 05:37:48,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41655.51 MB 2025-02-15 05:37:48,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:37:48,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:37:48,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 05:37:48,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:48,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32723.11 MB 2025-02-15 05:37:48,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25551.38 MB 2025-02-15 05:37:48,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7171.73 MB 2025-02-15 05:37:48,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39889.93 MB 2025-02-15 05:37:48,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54433.68 MB 2025-02-15 05:37:48,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14543.75 MB 2025-02-15 05:37:48,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52465.24 MB 2025-02-15 05:37:50,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:37:50,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:37:50,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:37:50,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25551.38 MB 2025-02-15 05:37:50,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26082.23 MB 2025-02-15 05:37:50,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:37:50,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54433.68 MB 2025-02-15 05:37:50,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-15 05:37:50,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23970.45 MB 2025-02-15 05:37:50,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30061.81 MB 2025-02-15 05:37:50,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:37:50,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:37:50,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:37:50,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26082.23 MB 2025-02-15 05:37:50,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27971.76 MB 2025-02-15 05:37:50,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:37:50,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 05:37:50,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-15 05:37:50,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:37:50,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29389.19 MB 2025-02-15 05:37:50,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:37:50,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:37:50,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:37:50,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27971.76 MB 2025-02-15 05:37:50,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30213.62 MB 2025-02-15 05:37:50,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:37:50,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 05:37:50,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37778.10 MB 2025-02-15 05:37:50,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 05:37:50,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35757.90 MB 2025-02-15 05:37:50,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:37:50,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:37:50,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:37:50,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26082.23 MB 2025-02-15 05:37:50,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30213.62 MB 2025-02-15 05:37:50,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:37:50,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-15 05:37:50,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37778.10 MB 2025-02-15 05:37:50,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 05:37:50,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35757.90 MB 2025-02-15 05:37:50,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:37:50,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:37:50,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:37:50,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31747.16 MB 2025-02-15 05:37:50,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32514.16 MB 2025-02-15 05:37:50,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:37:50,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37778.10 MB 2025-02-15 05:37:50,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38195.43 MB 2025-02-15 05:37:50,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:37:50,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33221.95 MB 2025-02-15 05:37:50,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:37:50,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:37:50,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:37:50,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32927.05 MB 2025-02-15 05:37:50,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33156.38 MB 2025-02-15 05:37:50,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.33 MB 2025-02-15 05:37:50,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38195.43 MB 2025-02-15 05:37:50,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38195.43 MB 2025-02-15 05:37:50,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:37:50,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33391.37 MB 2025-02-15 05:37:50,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:37:50,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:37:50,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.49 seconds 2025-02-15 05:37:50,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19518.78 MB 2025-02-15 05:37:50,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33357.40 MB 2025-02-15 05:37:50,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13838.63 MB 2025-02-15 05:37:50,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51428.46 MB 2025-02-15 05:37:50,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38195.43 MB 2025-02-15 05:37:50,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13233.03 MB 2025-02-15 05:37:50,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33391.37 MB 2025-02-15 05:37:50,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:37:50,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:37:50,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:37:50,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33357.40 MB 2025-02-15 05:37:50,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24522.40 MB 2025-02-15 05:37:50,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8835.00 MB 2025-02-15 05:37:50,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38195.43 MB 2025-02-15 05:37:50,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38195.43 MB 2025-02-15 05:37:50,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:37:50,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35868.46 MB 2025-02-15 05:37:50,794 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 05:37:50,794 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:37:50,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:37:50,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:37:50,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:37:50,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:37:50,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24522.40 MB 2025-02-15 05:37:50,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32959.88 MB 2025-02-15 05:37:50,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 05:37:50,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38195.43 MB 2025-02-15 05:37:50,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46584.04 MB 2025-02-15 05:37:50,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 05:37:50,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32959.88 MB 2025-02-15 05:37:50,962 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 05:37:50,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:50,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:37:50,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:50,964 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:37:50,969 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:37:50,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:50,970 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:37:50,970 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:37:59,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:59,337 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:37:59,342 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:37:59,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:59,346 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:37:59,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:37:59,347 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:38:30,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:38:30,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:38:30,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.32 seconds 2025-02-15 05:38:30,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:30,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-15 05:38:30,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-15 05:38:30,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-15 05:38:30,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54972.65 MB 2025-02-15 05:38:30,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40336.62 MB 2025-02-15 05:38:30,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14636.02 MB 2025-02-15 05:38:30,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42972.55 MB 2025-02-15 05:38:30,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:38:30,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:38:30,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:38:30,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:30,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-15 05:38:30,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24426.22 MB 2025-02-15 05:38:30,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9599.25 MB 2025-02-15 05:38:30,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40336.62 MB 2025-02-15 05:38:30,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40336.62 MB 2025-02-15 05:38:30,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:30,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36104.28 MB 2025-02-15 05:38:31,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:38:31,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:38:31,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 05:38:31,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.22 MB 2025-02-15 05:38:31,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24622.63 MB 2025-02-15 05:38:31,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-15 05:38:31,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40336.62 MB 2025-02-15 05:38:31,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:38:31,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7092.57 MB 2025-02-15 05:38:31,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28595.87 MB 2025-02-15 05:38:31,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:38:31,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:38:31,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:38:31,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-15 05:38:31,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25321.59 MB 2025-02-15 05:38:31,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-15 05:38:31,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:38:31,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:38:31,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:31,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25846.04 MB 2025-02-15 05:38:31,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:38:31,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:38:31,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:38:31,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25321.59 MB 2025-02-15 05:38:31,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-15 05:38:31,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-15 05:38:31,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:38:31,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:38:31,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:31,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-15 05:38:31,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:38:31,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:38:31,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:38:31,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-15 05:38:31,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-15 05:38:31,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-15 05:38:31,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:38:31,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:38:31,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:31,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-15 05:38:31,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:38:31,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:38:31,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:38:31,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26718.53 MB 2025-02-15 05:38:31,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.32 MB 2025-02-15 05:38:31,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-15 05:38:31,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:38:31,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33395.05 MB 2025-02-15 05:38:31,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 05:38:31,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27274.89 MB 2025-02-15 05:38:31,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:38:31,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:38:31,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:38:31,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27155.09 MB 2025-02-15 05:38:31,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27249.55 MB 2025-02-15 05:38:31,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.46 MB 2025-02-15 05:38:31,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33395.05 MB 2025-02-15 05:38:31,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33397.15 MB 2025-02-15 05:38:31,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 05:38:31,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27249.55 MB 2025-02-15 05:38:31,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:38:31,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:38:31,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.29 seconds 2025-02-15 05:38:31,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-15 05:38:31,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27332.89 MB 2025-02-15 05:38:31,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7382.09 MB 2025-02-15 05:38:31,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54972.65 MB 2025-02-15 05:38:31,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33397.15 MB 2025-02-15 05:38:31,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21575.50 MB 2025-02-15 05:38:31,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27332.89 MB 2025-02-15 05:38:31,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:38:31,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:38:31,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:38:31,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27332.89 MB 2025-02-15 05:38:31,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21942.32 MB 2025-02-15 05:38:31,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5390.57 MB 2025-02-15 05:38:31,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33397.15 MB 2025-02-15 05:38:31,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33397.15 MB 2025-02-15 05:38:31,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:31,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27790.98 MB 2025-02-15 05:38:31,758 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3375, cut from 3377 2025-02-15 05:38:31,758 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 1 ('] 2025-02-15 05:38:31,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:38:31,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:38:31,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:38:31,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:31,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21942.32 MB 2025-02-15 05:38:31,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25439.91 MB 2025-02-15 05:38:31,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3497.59 MB 2025-02-15 05:38:31,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33397.15 MB 2025-02-15 05:38:31,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33397.15 MB 2025-02-15 05:38:31,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:31,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25439.91 MB 2025-02-15 05:38:31,828 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3167] 2025-02-15 05:38:31,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:31,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:38:31,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:31,830 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:38:31,835 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:38:31,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:31,836 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:38:31,836 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 1 ('] 2025-02-15 05:38:52,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:52,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:38:52,904 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:38:52,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:52,911 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:38:52,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:52,912 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:38:55,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:38:55,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:38:55,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.31 seconds 2025-02-15 05:38:55,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-15 05:38:55,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-15 05:38:55,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-15 05:38:55,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33397.15 MB 2025-02-15 05:38:55,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12729.71 MB 2025-02-15 05:38:55,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.46 MB 2025-02-15 05:38:55,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:38:55,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:38:55,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:38:55,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-15 05:38:55,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14150.92 MB 2025-02-15 05:38:55,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -341.32 MB 2025-02-15 05:38:55,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15349.11 MB 2025-02-15 05:38:55,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:38:55,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:38:55,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 05:38:55,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14150.92 MB 2025-02-15 05:38:55,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14231.87 MB 2025-02-15 05:38:55,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-15 05:38:55,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18044.17 MB 2025-02-15 05:38:55,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:38:55,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:38:55,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:38:55,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-15 05:38:55,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14519.89 MB 2025-02-15 05:38:55,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-15 05:38:55,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14736.06 MB 2025-02-15 05:38:55,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:38:55,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:38:55,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:38:55,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14519.89 MB 2025-02-15 05:38:55,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14869.83 MB 2025-02-15 05:38:55,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-15 05:38:55,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.28 MB 2025-02-15 05:38:55,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:38:55,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:38:55,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:38:55,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-15 05:38:55,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14869.83 MB 2025-02-15 05:38:55,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.02 MB 2025-02-15 05:38:55,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-15 05:38:55,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.28 MB 2025-02-15 05:38:55,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:38:55,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:38:55,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:38:55,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15208.54 MB 2025-02-15 05:38:55,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15355.49 MB 2025-02-15 05:38:55,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-15 05:38:55,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-15 05:38:55,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20757.61 MB 2025-02-15 05:38:55,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-15 05:38:55,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15463.43 MB 2025-02-15 05:38:55,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:38:55,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:38:55,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:38:55,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15448.45 MB 2025-02-15 05:38:55,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15595.55 MB 2025-02-15 05:38:55,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.10 MB 2025-02-15 05:38:55,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20757.61 MB 2025-02-15 05:38:55,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20757.61 MB 2025-02-15 05:38:55,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15595.55 MB 2025-02-15 05:38:55,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:38:55,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:38:55,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.78 seconds 2025-02-15 05:38:55,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-15 05:38:55,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15727.39 MB 2025-02-15 05:38:55,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2253.49 MB 2025-02-15 05:38:55,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33397.15 MB 2025-02-15 05:38:55,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20757.61 MB 2025-02-15 05:38:55,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12639.54 MB 2025-02-15 05:38:55,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15727.39 MB 2025-02-15 05:38:55,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:38:55,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:38:55,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 05:38:55,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15727.39 MB 2025-02-15 05:38:55,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15806.07 MB 2025-02-15 05:38:55,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 78.68 MB 2025-02-15 05:38:55,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20757.61 MB 2025-02-15 05:38:55,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20757.61 MB 2025-02-15 05:38:55,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:38:55,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17637.79 MB 2025-02-15 05:38:55,906 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-15 05:38:55,907 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:38:55,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:38:55,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:38:55,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:38:55,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:38:55,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15806.07 MB 2025-02-15 05:38:55,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21339.99 MB 2025-02-15 05:38:55,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.92 MB 2025-02-15 05:38:55,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20757.61 MB 2025-02-15 05:38:55,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23509.07 MB 2025-02-15 05:38:55,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2751.46 MB 2025-02-15 05:38:55,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21339.99 MB 2025-02-15 05:38:56,078 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-15 05:38:56,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:56,080 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:38:56,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:56,082 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:38:56,090 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:38:56,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:38:56,092 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:38:56,092 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:39:30,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:30,262 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:39:30,270 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:39:30,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:30,277 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 427, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:39:30,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:30,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 427, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:39:36,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:39:36,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:39:36,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.68 seconds 2025-02-15 05:39:36,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:36,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15944.11 MB 2025-02-15 05:39:36,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17456.16 MB 2025-02-15 05:39:36,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1512.05 MB 2025-02-15 05:39:36,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29012.00 MB 2025-02-15 05:39:36,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20740.83 MB 2025-02-15 05:39:36,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8271.17 MB 2025-02-15 05:39:36,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26322.26 MB 2025-02-15 05:39:36,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:39:36,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:39:36,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:39:36,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:36,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17456.16 MB 2025-02-15 05:39:36,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17998.74 MB 2025-02-15 05:39:36,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 542.58 MB 2025-02-15 05:39:36,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20740.83 MB 2025-02-15 05:39:36,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25868.37 MB 2025-02-15 05:39:36,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5127.54 MB 2025-02-15 05:39:36,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24681.74 MB 2025-02-15 05:39:38,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:39:38,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:39:38,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:39:38,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:38,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17998.74 MB 2025-02-15 05:39:38,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18529.58 MB 2025-02-15 05:39:38,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:39:38,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25868.37 MB 2025-02-15 05:39:38,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21353.20 MB 2025-02-15 05:39:38,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4515.17 MB 2025-02-15 05:39:38,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22509.17 MB 2025-02-15 05:39:38,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:39:38,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:39:38,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:39:38,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:38,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18529.58 MB 2025-02-15 05:39:38,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20419.12 MB 2025-02-15 05:39:38,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:39:38,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21353.20 MB 2025-02-15 05:39:38,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23240.64 MB 2025-02-15 05:39:38,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 05:39:38,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21836.54 MB 2025-02-15 05:39:39,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:39:39,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:39:39,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:39:39,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20419.12 MB 2025-02-15 05:39:39,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22660.97 MB 2025-02-15 05:39:39,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:39:39,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23240.64 MB 2025-02-15 05:39:39,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30318.53 MB 2025-02-15 05:39:39,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 05:39:39,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28205.25 MB 2025-02-15 05:39:39,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:39:39,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:39:39,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:39:39,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18529.58 MB 2025-02-15 05:39:39,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22660.97 MB 2025-02-15 05:39:39,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:39:39,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21353.20 MB 2025-02-15 05:39:39,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30318.53 MB 2025-02-15 05:39:39,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 05:39:39,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28205.25 MB 2025-02-15 05:39:39,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:39:39,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:39:39,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:39:39,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24194.51 MB 2025-02-15 05:39:39,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24961.52 MB 2025-02-15 05:39:39,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:39:39,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30318.53 MB 2025-02-15 05:39:39,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30735.86 MB 2025-02-15 05:39:39,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:39:39,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25669.30 MB 2025-02-15 05:39:39,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:39:39,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:39:39,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:39:39,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25374.40 MB 2025-02-15 05:39:39,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25603.02 MB 2025-02-15 05:39:39,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-15 05:39:39,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30735.86 MB 2025-02-15 05:39:39,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30735.86 MB 2025-02-15 05:39:39,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:39:39,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25804.66 MB 2025-02-15 05:39:39,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:39:39,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:39:39,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.05 seconds 2025-02-15 05:39:39,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14456.41 MB 2025-02-15 05:39:39,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25803.70 MB 2025-02-15 05:39:39,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11347.29 MB 2025-02-15 05:39:39,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29012.00 MB 2025-02-15 05:39:39,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30735.86 MB 2025-02-15 05:39:39,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1723.86 MB 2025-02-15 05:39:39,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25804.66 MB 2025-02-15 05:39:39,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:39:39,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:39:39,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:39:39,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25803.70 MB 2025-02-15 05:39:39,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19454.88 MB 2025-02-15 05:39:39,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6348.83 MB 2025-02-15 05:39:39,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30735.86 MB 2025-02-15 05:39:39,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30735.86 MB 2025-02-15 05:39:39,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:39:39,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28310.45 MB 2025-02-15 05:39:39,615 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 05:39:39,615 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:39:39,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:39:39,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:39:39,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:39:39,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:39:39,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19454.88 MB 2025-02-15 05:39:39,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27877.20 MB 2025-02-15 05:39:39,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 05:39:39,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30735.86 MB 2025-02-15 05:39:39,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41204.84 MB 2025-02-15 05:39:39,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 05:39:39,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.20 MB 2025-02-15 05:39:39,779 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 05:39:39,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:39,781 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:39:39,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:39,782 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:39:39,786 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:39:39,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:39:39,788 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:39:39,788 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:40:35,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:35,153 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:40:35,161 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:40:35,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:35,168 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:40:35,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:35,170 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:40:45,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:40:45,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:40:45,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.07 seconds 2025-02-15 05:40:45,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:45,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.01 MB 2025-02-15 05:40:45,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19798.59 MB 2025-02-15 05:40:45,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2300.58 MB 2025-02-15 05:40:45,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53764.69 MB 2025-02-15 05:40:45,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25039.99 MB 2025-02-15 05:40:45,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28724.69 MB 2025-02-15 05:40:45,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28782.13 MB 2025-02-15 05:40:45,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:40:45,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:40:45,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 05:40:45,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:45,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19798.59 MB 2025-02-15 05:40:45,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19157.00 MB 2025-02-15 05:40:45,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -641.59 MB 2025-02-15 05:40:45,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25039.99 MB 2025-02-15 05:40:45,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30593.25 MB 2025-02-15 05:40:45,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5553.26 MB 2025-02-15 05:40:45,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28161.05 MB 2025-02-15 05:40:47,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:40:47,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:40:47,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:40:47,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19157.00 MB 2025-02-15 05:40:47,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.84 MB 2025-02-15 05:40:47,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:40:47,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30593.25 MB 2025-02-15 05:40:47,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22059.94 MB 2025-02-15 05:40:47,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8533.31 MB 2025-02-15 05:40:47,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23667.43 MB 2025-02-15 05:40:47,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:40:47,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:40:47,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:40:47,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-15 05:40:47,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21577.37 MB 2025-02-15 05:40:47,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:40:47,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22059.94 MB 2025-02-15 05:40:47,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24891.10 MB 2025-02-15 05:40:47,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:40:47,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22994.80 MB 2025-02-15 05:40:47,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:40:47,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:40:47,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:40:47,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21577.37 MB 2025-02-15 05:40:47,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-15 05:40:47,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:40:47,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24891.10 MB 2025-02-15 05:40:47,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31499.22 MB 2025-02-15 05:40:47,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 05:40:47,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29364.56 MB 2025-02-15 05:40:47,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:40:47,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:40:47,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:40:47,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-15 05:40:47,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-15 05:40:47,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:40:47,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22059.94 MB 2025-02-15 05:40:47,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31499.22 MB 2025-02-15 05:40:47,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-15 05:40:47,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29364.56 MB 2025-02-15 05:40:47,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:40:47,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:40:47,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:40:47,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25353.82 MB 2025-02-15 05:40:47,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26120.82 MB 2025-02-15 05:40:47,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:40:47,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31499.22 MB 2025-02-15 05:40:47,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-15 05:40:47,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:40:47,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26828.61 MB 2025-02-15 05:40:47,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:40:47,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:40:47,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:40:47,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26533.71 MB 2025-02-15 05:40:47,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26765.08 MB 2025-02-15 05:40:47,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.37 MB 2025-02-15 05:40:47,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31916.56 MB 2025-02-15 05:40:47,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-15 05:40:47,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:40:47,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26976.83 MB 2025-02-15 05:40:47,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:40:47,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:40:47,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.46 seconds 2025-02-15 05:40:47,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15233.36 MB 2025-02-15 05:40:47,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26966.15 MB 2025-02-15 05:40:47,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11732.80 MB 2025-02-15 05:40:47,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53764.69 MB 2025-02-15 05:40:47,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-15 05:40:47,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21848.13 MB 2025-02-15 05:40:47,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26976.83 MB 2025-02-15 05:40:47,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:40:47,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:40:47,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:40:47,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26966.15 MB 2025-02-15 05:40:47,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20239.16 MB 2025-02-15 05:40:47,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6726.99 MB 2025-02-15 05:40:47,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31916.56 MB 2025-02-15 05:40:47,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-15 05:40:47,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:40:47,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29478.18 MB 2025-02-15 05:40:47,917 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:40:47,917 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:40:47,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:40:47,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:40:47,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:40:47,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:40:47,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20239.16 MB 2025-02-15 05:40:47,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28678.18 MB 2025-02-15 05:40:47,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:40:47,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31916.56 MB 2025-02-15 05:40:47,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40307.26 MB 2025-02-15 05:40:47,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:40:47,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28678.18 MB 2025-02-15 05:40:48,084 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:40:48,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:48,086 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:40:48,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:48,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:40:48,091 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:40:48,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:48,092 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:40:48,093 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:40:58,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:58,833 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:40:58,838 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:40:58,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:58,841 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1160, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:40:58,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:40:58,842 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1160, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:41:16,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:41:16,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:41:16,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.02 seconds 2025-02-15 05:41:16,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:16,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21051.77 MB 2025-02-15 05:41:16,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25158.00 MB 2025-02-15 05:41:16,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4106.22 MB 2025-02-15 05:41:16,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52892.27 MB 2025-02-15 05:41:16,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27529.31 MB 2025-02-15 05:41:16,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25362.96 MB 2025-02-15 05:41:16,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34147.83 MB 2025-02-15 05:41:16,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:41:16,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:41:16,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:41:16,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:16,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25158.00 MB 2025-02-15 05:41:16,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21809.38 MB 2025-02-15 05:41:16,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3348.62 MB 2025-02-15 05:41:16,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27529.31 MB 2025-02-15 05:41:16,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37325.11 MB 2025-02-15 05:41:16,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9795.80 MB 2025-02-15 05:41:16,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37053.90 MB 2025-02-15 05:41:18,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:41:18,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:41:18,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 05:41:18,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:18,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21809.38 MB 2025-02-15 05:41:18,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22340.22 MB 2025-02-15 05:41:18,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:41:18,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37325.11 MB 2025-02-15 05:41:18,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25547.51 MB 2025-02-15 05:41:18,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11777.61 MB 2025-02-15 05:41:18,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26319.80 MB 2025-02-15 05:41:18,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:41:18,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:41:18,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:41:18,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:18,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22340.22 MB 2025-02-15 05:41:18,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24229.75 MB 2025-02-15 05:41:18,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:41:18,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25547.51 MB 2025-02-15 05:41:18,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27434.94 MB 2025-02-15 05:41:18,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 05:41:18,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25647.18 MB 2025-02-15 05:41:19,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:41:19,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:41:19,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:41:19,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24229.75 MB 2025-02-15 05:41:19,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26472.66 MB 2025-02-15 05:41:19,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 05:41:19,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27434.94 MB 2025-02-15 05:41:19,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-15 05:41:19,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 05:41:19,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32016.94 MB 2025-02-15 05:41:19,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:41:19,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:41:19,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:41:19,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22340.22 MB 2025-02-15 05:41:19,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26472.66 MB 2025-02-15 05:41:19,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 05:41:19,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25547.51 MB 2025-02-15 05:41:19,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-15 05:41:19,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-15 05:41:19,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32016.94 MB 2025-02-15 05:41:19,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:41:19,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:41:19,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:41:19,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28006.20 MB 2025-02-15 05:41:19,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28773.20 MB 2025-02-15 05:41:19,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:41:19,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34043.07 MB 2025-02-15 05:41:19,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-15 05:41:19,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:41:19,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29480.99 MB 2025-02-15 05:41:19,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:41:19,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:41:19,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:41:19,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29186.09 MB 2025-02-15 05:41:19,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29414.02 MB 2025-02-15 05:41:19,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.93 MB 2025-02-15 05:41:19,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-15 05:41:19,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-15 05:41:19,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:41:19,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29614.66 MB 2025-02-15 05:41:19,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:41:19,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:41:19,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.50 seconds 2025-02-15 05:41:19,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17010.24 MB 2025-02-15 05:41:19,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29614.87 MB 2025-02-15 05:41:19,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12604.63 MB 2025-02-15 05:41:19,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52892.27 MB 2025-02-15 05:41:19,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-15 05:41:19,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18431.87 MB 2025-02-15 05:41:19,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29614.87 MB 2025-02-15 05:41:19,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:41:19,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:41:19,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:41:19,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29614.87 MB 2025-02-15 05:41:19,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22000.15 MB 2025-02-15 05:41:19,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7614.72 MB 2025-02-15 05:41:19,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-15 05:41:19,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-15 05:41:19,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:41:19,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32114.25 MB 2025-02-15 05:41:19,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 05:41:19,626 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:41:19,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:41:19,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:41:19,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:41:19,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:41:19,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22000.15 MB 2025-02-15 05:41:19,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30397.56 MB 2025-02-15 05:41:19,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 05:41:19,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-15 05:41:19,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42811.26 MB 2025-02-15 05:41:19,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 05:41:19,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30397.56 MB 2025-02-15 05:41:19,792 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 05:41:19,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:41:19,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:41:19,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:41:19,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:41:19,799 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:41:19,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:41:19,800 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:41:19,800 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:42:21,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:21,531 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:42:21,536 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:42:21,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:21,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 196, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:42:21,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:21,541 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 196, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:42:24,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:42:24,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:42:24,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-15 05:42:24,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:24,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14334.47 MB 2025-02-15 05:42:24,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15028.10 MB 2025-02-15 05:42:24,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 693.63 MB 2025-02-15 05:42:24,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51162.12 MB 2025-02-15 05:42:24,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 05:42:24,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32558.28 MB 2025-02-15 05:42:24,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24032.33 MB 2025-02-15 05:42:24,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:42:24,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:42:24,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:42:24,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:24,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15028.10 MB 2025-02-15 05:42:24,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15237.75 MB 2025-02-15 05:42:24,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.65 MB 2025-02-15 05:42:24,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 05:42:24,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19235.08 MB 2025-02-15 05:42:24,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 631.24 MB 2025-02-15 05:42:24,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17528.36 MB 2025-02-15 05:42:25,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:42:25,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:42:25,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 05:42:25,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15237.75 MB 2025-02-15 05:42:25,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15473.97 MB 2025-02-15 05:42:25,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-15 05:42:25,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19235.08 MB 2025-02-15 05:42:25,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 05:42:25,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -157.29 MB 2025-02-15 05:42:25,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19408.44 MB 2025-02-15 05:42:25,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:42:25,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:42:25,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:42:25,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-15 05:42:25,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16314.55 MB 2025-02-15 05:42:25,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-15 05:42:25,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 05:42:25,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 05:42:25,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:42:25,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16945.31 MB 2025-02-15 05:42:25,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:42:25,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:42:25,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:42:25,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16314.55 MB 2025-02-15 05:42:25,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-15 05:42:25,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-15 05:42:25,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 05:42:25,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21187.53 MB 2025-02-15 05:42:25,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2109.73 MB 2025-02-15 05:42:25,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-15 05:42:25,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:42:25,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:42:25,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:42:25,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-15 05:42:25,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-15 05:42:25,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-15 05:42:25,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 05:42:25,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21187.53 MB 2025-02-15 05:42:25,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2109.73 MB 2025-02-15 05:42:25,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-15 05:42:25,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:42:25,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:42:25,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:42:25,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17994.63 MB 2025-02-15 05:42:25,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.95 MB 2025-02-15 05:42:25,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.32 MB 2025-02-15 05:42:25,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21187.53 MB 2025-02-15 05:42:25,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21369.98 MB 2025-02-15 05:42:25,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 05:42:25,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18658.07 MB 2025-02-15 05:42:25,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:42:25,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:42:25,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:42:25,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18519.69 MB 2025-02-15 05:42:25,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18724.22 MB 2025-02-15 05:42:25,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.52 MB 2025-02-15 05:42:25,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21369.98 MB 2025-02-15 05:42:25,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-15 05:42:25,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 05:42:25,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18760.79 MB 2025-02-15 05:42:25,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:42:25,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:42:25,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.10 seconds 2025-02-15 05:42:25,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13651.59 MB 2025-02-15 05:42:25,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18925.29 MB 2025-02-15 05:42:25,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5273.70 MB 2025-02-15 05:42:25,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51162.12 MB 2025-02-15 05:42:25,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-15 05:42:25,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29787.95 MB 2025-02-15 05:42:25,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18925.29 MB 2025-02-15 05:42:25,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:42:25,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:42:25,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:42:25,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18925.29 MB 2025-02-15 05:42:25,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17608.03 MB 2025-02-15 05:42:25,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1317.26 MB 2025-02-15 05:42:25,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21374.17 MB 2025-02-15 05:42:25,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-15 05:42:25,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:42:25,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19126.26 MB 2025-02-15 05:42:25,924 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:42:25,924 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:42:25,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:42:25,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:42:25,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:42:25,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:42:25,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17608.03 MB 2025-02-15 05:42:25,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26047.05 MB 2025-02-15 05:42:25,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:42:25,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21374.17 MB 2025-02-15 05:42:25,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31864.13 MB 2025-02-15 05:42:25,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:42:25,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26047.05 MB 2025-02-15 05:42:26,088 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:42:26,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:26,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:42:26,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:26,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:42:26,095 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:42:26,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:42:26,096 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:42:26,096 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:43:16,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:16,938 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:43:16,944 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:43:16,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:16,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1308, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:43:16,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:16,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1308, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:43:37,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:43:37,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:43:37,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.14 seconds 2025-02-15 05:43:37,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:37,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22083.06 MB 2025-02-15 05:43:37,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26712.00 MB 2025-02-15 05:43:37,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4628.94 MB 2025-02-15 05:43:37,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44449.14 MB 2025-02-15 05:43:37,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37889.25 MB 2025-02-15 05:43:37,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6559.89 MB 2025-02-15 05:43:37,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35631.29 MB 2025-02-15 05:43:37,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:43:37,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:43:37,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:43:37,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:37,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26712.00 MB 2025-02-15 05:43:37,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22577.73 MB 2025-02-15 05:43:37,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4134.26 MB 2025-02-15 05:43:37,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37889.25 MB 2025-02-15 05:43:37,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44379.93 MB 2025-02-15 05:43:37,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6490.69 MB 2025-02-15 05:43:37,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39618.76 MB 2025-02-15 05:43:39,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:43:39,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:43:39,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:43:39,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22577.73 MB 2025-02-15 05:43:39,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23108.58 MB 2025-02-15 05:43:39,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:43:39,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44379.93 MB 2025-02-15 05:43:39,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:43:39,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15315.50 MB 2025-02-15 05:43:39,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27087.12 MB 2025-02-15 05:43:39,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:43:39,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:43:39,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:43:39,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23108.58 MB 2025-02-15 05:43:39,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24998.11 MB 2025-02-15 05:43:39,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:43:39,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:43:39,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:43:39,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:43:39,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26415.54 MB 2025-02-15 05:43:39,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:43:39,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:43:39,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:43:39,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24998.11 MB 2025-02-15 05:43:39,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27239.97 MB 2025-02-15 05:43:39,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:43:39,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:43:39,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:43:39,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:43:39,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32784.25 MB 2025-02-15 05:43:39,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:43:39,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:43:39,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:43:39,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23108.58 MB 2025-02-15 05:43:39,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27239.97 MB 2025-02-15 05:43:39,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:43:39,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:43:39,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:43:39,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:43:39,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32784.25 MB 2025-02-15 05:43:39,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:43:39,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:43:39,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:43:39,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28773.51 MB 2025-02-15 05:43:39,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29540.51 MB 2025-02-15 05:43:39,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:43:39,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34726.74 MB 2025-02-15 05:43:39,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:43:39,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:43:39,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30248.30 MB 2025-02-15 05:43:39,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:43:39,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:43:39,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:43:39,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29953.40 MB 2025-02-15 05:43:39,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30181.73 MB 2025-02-15 05:43:39,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.33 MB 2025-02-15 05:43:39,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:43:39,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:43:39,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:43:39,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30389.70 MB 2025-02-15 05:43:39,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:43:39,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:43:39,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.56 seconds 2025-02-15 05:43:39,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17525.88 MB 2025-02-15 05:43:39,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30382.58 MB 2025-02-15 05:43:39,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12856.70 MB 2025-02-15 05:43:39,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44449.14 MB 2025-02-15 05:43:39,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:43:39,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9307.16 MB 2025-02-15 05:43:39,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30389.70 MB 2025-02-15 05:43:39,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:43:39,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:43:39,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:43:39,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30382.58 MB 2025-02-15 05:43:39,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22526.84 MB 2025-02-15 05:43:39,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7855.74 MB 2025-02-15 05:43:39,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:43:39,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:43:39,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:43:39,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32891.49 MB 2025-02-15 05:43:39,794 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 05:43:39,794 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:43:39,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:43:39,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:43:39,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:43:39,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:43:39,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22526.84 MB 2025-02-15 05:43:39,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30956.49 MB 2025-02-15 05:43:39,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.64 MB 2025-02-15 05:43:39,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:43:39,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-15 05:43:39,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 05:43:39,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30956.49 MB 2025-02-15 05:43:39,962 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 05:43:39,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:39,964 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:43:39,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:39,965 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:43:39,969 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:43:39,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:43:39,970 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:43:39,970 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:44:36,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:44:36,733 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:44:36,739 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:44:36,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:44:36,743 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1466, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:44:36,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:44:36,745 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1466, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:44:59,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:44:59,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:44:59,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.58 seconds 2025-02-15 05:44:59,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:44:59,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23184.03 MB 2025-02-15 05:44:59,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28372.38 MB 2025-02-15 05:44:59,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5188.35 MB 2025-02-15 05:44:59,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47712.31 MB 2025-02-15 05:44:59,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38432.41 MB 2025-02-15 05:44:59,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9279.90 MB 2025-02-15 05:44:59,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37185.25 MB 2025-02-15 05:44:59,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:44:59,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:44:59,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:44:59,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:44:59,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28372.38 MB 2025-02-15 05:44:59,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23399.13 MB 2025-02-15 05:44:59,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4973.26 MB 2025-02-15 05:44:59,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38432.41 MB 2025-02-15 05:44:59,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47796.19 MB 2025-02-15 05:44:59,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9363.78 MB 2025-02-15 05:44:59,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42452.69 MB 2025-02-15 05:45:01,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:45:01,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:45:01,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:45:01,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23399.13 MB 2025-02-15 05:45:01,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23929.97 MB 2025-02-15 05:45:01,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:45:01,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47796.19 MB 2025-02-15 05:45:01,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:45:01,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14552.14 MB 2025-02-15 05:45:01,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27908.52 MB 2025-02-15 05:45:01,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:45:01,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:45:01,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:45:01,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23929.97 MB 2025-02-15 05:45:01,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25819.50 MB 2025-02-15 05:45:01,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:45:01,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:45:01,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 05:45:01,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:01,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27236.93 MB 2025-02-15 05:45:01,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:45:01,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:45:01,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:45:01,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25819.50 MB 2025-02-15 05:45:01,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28061.36 MB 2025-02-15 05:45:01,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:45:01,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:45:01,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36075.21 MB 2025-02-15 05:45:01,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:45:01,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33605.64 MB 2025-02-15 05:45:01,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:45:01,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:45:01,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 05:45:01,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23929.97 MB 2025-02-15 05:45:01,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28061.36 MB 2025-02-15 05:45:01,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:45:01,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 05:45:01,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36075.21 MB 2025-02-15 05:45:01,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:45:01,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33605.64 MB 2025-02-15 05:45:01,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:45:01,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:45:01,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:45:01,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29594.90 MB 2025-02-15 05:45:01,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30361.90 MB 2025-02-15 05:45:01,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:45:01,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36075.21 MB 2025-02-15 05:45:01,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-15 05:45:01,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:45:01,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31069.69 MB 2025-02-15 05:45:01,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:45:01,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:45:01,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:45:01,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30774.79 MB 2025-02-15 05:45:01,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31003.02 MB 2025-02-15 05:45:01,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 05:45:01,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-15 05:45:01,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-15 05:45:01,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:01,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31247.19 MB 2025-02-15 05:45:01,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:45:01,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:45:01,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.03 seconds 2025-02-15 05:45:01,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:01,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18076.37 MB 2025-02-15 05:45:01,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31203.15 MB 2025-02-15 05:45:01,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13126.79 MB 2025-02-15 05:45:01,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47712.31 MB 2025-02-15 05:45:01,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-15 05:45:01,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11221.86 MB 2025-02-15 05:45:01,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31247.19 MB 2025-02-15 05:45:02,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:45:02,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:45:02,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 05:45:02,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:02,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31203.15 MB 2025-02-15 05:45:02,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23066.99 MB 2025-02-15 05:45:02,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8136.16 MB 2025-02-15 05:45:02,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-15 05:45:02,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-15 05:45:02,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:02,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33703.15 MB 2025-02-15 05:45:02,076 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 05:45:02,076 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:45:02,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:45:02,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:45:02,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:45:02,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:02,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23066.99 MB 2025-02-15 05:45:02,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31467.85 MB 2025-02-15 05:45:02,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 05:45:02,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-15 05:45:02,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44841.30 MB 2025-02-15 05:45:02,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 05:45:02,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31467.85 MB 2025-02-15 05:45:02,337 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 05:45:02,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:02,339 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:45:02,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:02,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:45:02,349 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:45:02,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:02,351 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:45:02,351 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:45:10,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:10,815 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:45:10,820 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:45:10,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:10,823 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:45:10,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:10,824 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:45:30,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:45:30,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:45:30,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.02 seconds 2025-02-15 05:45:30,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:30,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-15 05:45:30,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-15 05:45:30,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-15 05:45:30,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53192.16 MB 2025-02-15 05:45:30,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37763.42 MB 2025-02-15 05:45:30,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15428.75 MB 2025-02-15 05:45:30,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.03 MB 2025-02-15 05:45:30,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:45:30,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:45:30,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:45:30,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:30,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-15 05:45:30,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-15 05:45:30,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-15 05:45:30,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37763.42 MB 2025-02-15 05:45:30,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46762.30 MB 2025-02-15 05:45:30,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8998.88 MB 2025-02-15 05:45:30,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40001.99 MB 2025-02-15 05:45:32,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:45:32,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:45:32,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:45:32,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:32,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-15 05:45:32,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-15 05:45:32,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:45:32,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46762.30 MB 2025-02-15 05:45:32,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33214.69 MB 2025-02-15 05:45:32,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13547.60 MB 2025-02-15 05:45:32,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26967.55 MB 2025-02-15 05:45:32,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:45:32,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:45:32,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:45:32,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:32,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-15 05:45:32,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-15 05:45:32,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:45:32,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33214.69 MB 2025-02-15 05:45:32,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33214.69 MB 2025-02-15 05:45:32,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:32,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-15 05:45:33,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:45:33,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:45:33,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:45:33,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-15 05:45:33,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-15 05:45:33,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:45:33,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33214.69 MB 2025-02-15 05:45:33,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34158.41 MB 2025-02-15 05:45:33,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 05:45:33,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-15 05:45:33,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:45:33,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:45:33,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:45:33,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-15 05:45:33,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-15 05:45:33,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:45:33,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33214.69 MB 2025-02-15 05:45:33,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34158.41 MB 2025-02-15 05:45:33,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 05:45:33,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-15 05:45:33,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:45:33,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:45:33,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:45:33,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-15 05:45:33,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-15 05:45:33,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:45:33,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34158.41 MB 2025-02-15 05:45:33,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 05:45:33,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:45:33,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-15 05:45:33,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:45:33,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:45:33,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:45:33,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-15 05:45:33,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30061.52 MB 2025-02-15 05:45:33,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.70 MB 2025-02-15 05:45:33,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 05:45:33,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 05:45:33,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:33,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30302.00 MB 2025-02-15 05:45:33,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:45:33,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:45:33,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.44 seconds 2025-02-15 05:45:33,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-15 05:45:33,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30262.38 MB 2025-02-15 05:45:33,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12816.63 MB 2025-02-15 05:45:33,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53192.16 MB 2025-02-15 05:45:33,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 05:45:33,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18618.52 MB 2025-02-15 05:45:33,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30302.00 MB 2025-02-15 05:45:33,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:45:33,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:45:33,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:45:33,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30262.38 MB 2025-02-15 05:45:33,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22439.23 MB 2025-02-15 05:45:33,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7823.15 MB 2025-02-15 05:45:33,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 05:45:33,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 05:45:33,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:45:33,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32764.83 MB 2025-02-15 05:45:33,548 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 05:45:33,549 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:45:33,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:45:33,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:45:33,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:45:33,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:45:33,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22439.23 MB 2025-02-15 05:45:33,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30848.53 MB 2025-02-15 05:45:33,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 05:45:33,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 05:45:33,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42932.90 MB 2025-02-15 05:45:33,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 05:45:33,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30848.53 MB 2025-02-15 05:45:33,712 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 05:45:33,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:33,713 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:45:33,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:33,714 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:45:33,719 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:45:33,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:45:33,720 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:45:33,720 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:47:01,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:01,319 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:47:01,324 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:47:01,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:01,328 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 168, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:47:01,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:01,329 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 168, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:47:03,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:47:03,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:47:03,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.59 seconds 2025-02-15 05:47:03,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:03,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.36 MB 2025-02-15 05:47:03,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14733.90 MB 2025-02-15 05:47:03,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.54 MB 2025-02-15 05:47:03,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51292.14 MB 2025-02-15 05:47:03,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-15 05:47:03,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32233.23 MB 2025-02-15 05:47:03,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23610.73 MB 2025-02-15 05:47:03,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:47:03,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:47:03,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:47:03,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:03,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14733.90 MB 2025-02-15 05:47:03,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14979.82 MB 2025-02-15 05:47:03,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.92 MB 2025-02-15 05:47:03,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-15 05:47:03,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-15 05:47:03,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:47:03,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17009.42 MB 2025-02-15 05:47:04,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:47:04,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:47:04,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 05:47:04,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14979.82 MB 2025-02-15 05:47:04,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15194.81 MB 2025-02-15 05:47:04,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-15 05:47:04,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-15 05:47:04,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 05:47:04,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 05:47:04,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19150.50 MB 2025-02-15 05:47:04,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:47:04,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:47:04,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:47:04,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-15 05:47:04,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15959.82 MB 2025-02-15 05:47:04,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-15 05:47:04,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 05:47:04,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17804.82 MB 2025-02-15 05:47:04,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 383.78 MB 2025-02-15 05:47:04,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16533.88 MB 2025-02-15 05:47:04,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:47:04,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:47:04,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:47:04,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15959.82 MB 2025-02-15 05:47:04,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-15 05:47:04,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-15 05:47:04,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17804.82 MB 2025-02-15 05:47:04,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20300.43 MB 2025-02-15 05:47:04,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2495.61 MB 2025-02-15 05:47:04,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-15 05:47:04,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:47:04,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:47:04,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:47:04,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-15 05:47:04,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-15 05:47:04,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-15 05:47:04,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 05:47:04,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20300.43 MB 2025-02-15 05:47:04,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2879.39 MB 2025-02-15 05:47:04,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-15 05:47:04,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:47:04,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:47:04,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:47:04,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17488.89 MB 2025-02-15 05:47:04,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17799.53 MB 2025-02-15 05:47:04,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.64 MB 2025-02-15 05:47:04,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20300.43 MB 2025-02-15 05:47:04,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20468.20 MB 2025-02-15 05:47:04,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 05:47:04,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18094.24 MB 2025-02-15 05:47:04,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:47:04,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:47:04,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:47:04,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17966.76 MB 2025-02-15 05:47:04,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18171.08 MB 2025-02-15 05:47:04,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.33 MB 2025-02-15 05:47:04,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20468.20 MB 2025-02-15 05:47:04,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20472.40 MB 2025-02-15 05:47:04,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 05:47:04,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18196.35 MB 2025-02-15 05:47:04,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:47:04,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:47:04,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.56 seconds 2025-02-15 05:47:04,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:04,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13554.03 MB 2025-02-15 05:47:04,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18371.94 MB 2025-02-15 05:47:04,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4817.90 MB 2025-02-15 05:47:04,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51292.14 MB 2025-02-15 05:47:04,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20472.40 MB 2025-02-15 05:47:04,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30819.75 MB 2025-02-15 05:47:04,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18371.94 MB 2025-02-15 05:47:05,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:47:05,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:47:05,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:47:05,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:05,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18371.94 MB 2025-02-15 05:47:05,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17430.21 MB 2025-02-15 05:47:05,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -941.73 MB 2025-02-15 05:47:05,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20472.40 MB 2025-02-15 05:47:05,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20472.40 MB 2025-02-15 05:47:05,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:47:05,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19074.17 MB 2025-02-15 05:47:05,172 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 05:47:05,172 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:47:05,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:47:05,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:47:05,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:47:05,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:05,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17430.21 MB 2025-02-15 05:47:05,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25856.71 MB 2025-02-15 05:47:05,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 05:47:05,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20472.40 MB 2025-02-15 05:47:05,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30945.57 MB 2025-02-15 05:47:05,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 05:47:05,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25856.71 MB 2025-02-15 05:47:05,337 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 05:47:05,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:05,338 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:47:05,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:05,339 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:47:05,344 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:47:05,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:05,345 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:47:05,345 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:47:16,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:16,062 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:47:16,067 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:47:16,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:16,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:47:16,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:16,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:47:46,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:47:46,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:47:46,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.29 seconds 2025-02-15 05:47:46,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:46,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-15 05:47:46,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-15 05:47:46,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-15 05:47:46,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43511.71 MB 2025-02-15 05:47:46,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40170.95 MB 2025-02-15 05:47:46,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3340.76 MB 2025-02-15 05:47:46,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42439.46 MB 2025-02-15 05:47:46,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:47:46,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:47:46,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:47:46,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:46,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-15 05:47:46,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-15 05:47:46,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-15 05:47:46,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40170.95 MB 2025-02-15 05:47:46,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54905.54 MB 2025-02-15 05:47:46,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14734.59 MB 2025-02-15 05:47:46,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53591.85 MB 2025-02-15 05:47:48,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:47:48,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:47:48,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:47:48,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-15 05:47:48,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-15 05:47:48,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:47:48,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54905.54 MB 2025-02-15 05:47:48,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34649.15 MB 2025-02-15 05:47:48,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20256.39 MB 2025-02-15 05:47:48,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30476.67 MB 2025-02-15 05:47:48,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:47:48,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:47:48,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:47:48,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-15 05:47:48,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-15 05:47:48,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:47:48,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 05:47:48,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34649.15 MB 2025-02-15 05:47:48,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:47:48,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-15 05:47:48,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:47:48,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:47:48,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:47:48,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-15 05:47:48,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-15 05:47:48,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:47:48,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 05:47:48,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39367.74 MB 2025-02-15 05:47:48,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:47:48,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-15 05:47:48,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:47:48,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:47:48,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:47:48,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-15 05:47:48,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-15 05:47:48,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:47:48,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 05:47:48,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39367.74 MB 2025-02-15 05:47:48,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 05:47:48,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-15 05:47:48,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:47:48,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:47:48,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:47:48,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-15 05:47:48,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-15 05:47:48,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:47:48,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39367.74 MB 2025-02-15 05:47:48,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 05:47:48,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:47:48,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-15 05:47:48,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:47:48,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:47:48,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:47:48,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-15 05:47:48,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33571.12 MB 2025-02-15 05:47:48,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 05:47:48,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 05:47:48,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 05:47:48,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:47:48,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.55 MB 2025-02-15 05:47:48,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:47:48,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:47:48,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.81 seconds 2025-02-15 05:47:48,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:48,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-15 05:47:48,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33771.97 MB 2025-02-15 05:47:48,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13974.47 MB 2025-02-15 05:47:48,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43511.71 MB 2025-02-15 05:47:48,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 05:47:48,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3728.74 MB 2025-02-15 05:47:48,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.55 MB 2025-02-15 05:47:49,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:47:49,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:47:49,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:47:49,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:49,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33771.97 MB 2025-02-15 05:47:49,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.42 MB 2025-02-15 05:47:49,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8984.55 MB 2025-02-15 05:47:49,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 05:47:49,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39782.97 MB 2025-02-15 05:47:49,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:47:49,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36271.35 MB 2025-02-15 05:47:49,168 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 05:47:49,169 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:47:49,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:47:49,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:47:49,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:47:49,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:47:49,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.42 MB 2025-02-15 05:47:49,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33184.82 MB 2025-02-15 05:47:49,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 05:47:49,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39782.97 MB 2025-02-15 05:47:49,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48133.83 MB 2025-02-15 05:47:49,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 05:47:49,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.82 MB 2025-02-15 05:47:49,331 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 05:47:49,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:49,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:47:49,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:49,333 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:47:49,338 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:47:49,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:47:49,339 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:47:49,339 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:48:04,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:04,751 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:48:04,756 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:48:04,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:04,760 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:48:04,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:04,761 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:48:07,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:48:07,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:48:07,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-15 05:48:07,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:07,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-15 05:48:07,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-15 05:48:07,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-15 05:48:07,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56484.69 MB 2025-02-15 05:48:07,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:07,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34871.44 MB 2025-02-15 05:48:07,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.07 MB 2025-02-15 05:48:07,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:48:07,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:48:07,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:48:07,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:07,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-15 05:48:07,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15384.68 MB 2025-02-15 05:48:07,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.53 MB 2025-02-15 05:48:07,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:07,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:07,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:07,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17823.11 MB 2025-02-15 05:48:08,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:48:08,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:48:08,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 05:48:08,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:08,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15384.68 MB 2025-02-15 05:48:08,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15640.81 MB 2025-02-15 05:48:08,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 05:48:08,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:08,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:08,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:08,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19639.27 MB 2025-02-15 05:48:08,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:48:08,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:48:08,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:48:08,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:08,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 05:48:08,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.23 MB 2025-02-15 05:48:08,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 05:48:08,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:08,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:08,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:08,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.14 MB 2025-02-15 05:48:09,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:48:09,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:48:09,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:48:09,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.23 MB 2025-02-15 05:48:09,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 05:48:09,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 05:48:09,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:09,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:09,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:09,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 05:48:09,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:48:09,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:48:09,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:48:09,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 05:48:09,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 05:48:09,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 05:48:09,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:09,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21613.25 MB 2025-02-15 05:48:09,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:09,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 05:48:09,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:48:09,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:48:09,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:48:09,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.89 MB 2025-02-15 05:48:09,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18743.97 MB 2025-02-15 05:48:09,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 05:48:09,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21613.25 MB 2025-02-15 05:48:09,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 05:48:09,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-15 05:48:09,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19088.77 MB 2025-02-15 05:48:09,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:48:09,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:48:09,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:48:09,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18943.20 MB 2025-02-15 05:48:09,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19170.67 MB 2025-02-15 05:48:09,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.47 MB 2025-02-15 05:48:09,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-15 05:48:09,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 05:48:09,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:09,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19220.33 MB 2025-02-15 05:48:09,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:48:09,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:48:09,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.38 seconds 2025-02-15 05:48:09,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-15 05:48:09,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19371.74 MB 2025-02-15 05:48:09,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5692.28 MB 2025-02-15 05:48:09,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56484.69 MB 2025-02-15 05:48:09,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21812.48 MB 2025-02-15 05:48:09,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34672.21 MB 2025-02-15 05:48:09,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19371.74 MB 2025-02-15 05:48:09,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:48:09,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:48:09,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:48:09,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19371.74 MB 2025-02-15 05:48:09,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17706.95 MB 2025-02-15 05:48:09,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1664.79 MB 2025-02-15 05:48:09,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21812.48 MB 2025-02-15 05:48:09,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21812.48 MB 2025-02-15 05:48:09,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:48:09,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19371.74 MB 2025-02-15 05:48:09,428 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:48:09,428 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:48:09,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:48:09,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:48:09,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:48:09,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:48:09,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17706.95 MB 2025-02-15 05:48:09,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26145.98 MB 2025-02-15 05:48:09,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:48:09,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21812.48 MB 2025-02-15 05:48:09,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32302.43 MB 2025-02-15 05:48:09,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:48:09,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26145.98 MB 2025-02-15 05:48:09,592 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:48:09,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:09,594 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:48:09,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:09,595 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:48:09,599 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:48:09,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:48:09,600 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:48:09,600 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:49:01,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:01,983 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:49:01,990 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:49:01,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:01,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 294, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:49:01,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:01,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 294, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:49:06,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:49:06,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:49:06,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.53 seconds 2025-02-15 05:49:06,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15017.35 MB 2025-02-15 05:49:06,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16057.80 MB 2025-02-15 05:49:06,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1040.45 MB 2025-02-15 05:49:06,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44887.44 MB 2025-02-15 05:49:06,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25365.05 MB 2025-02-15 05:49:06,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24941.70 MB 2025-02-15 05:49:06,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:49:06,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:49:06,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:49:06,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16057.80 MB 2025-02-15 05:49:06,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14960.64 MB 2025-02-15 05:49:06,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1097.15 MB 2025-02-15 05:49:06,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16984.90 MB 2025-02-15 05:49:06,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:49:06,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:49:06,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.33 seconds 2025-02-15 05:49:06,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14960.64 MB 2025-02-15 05:49:06,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15048.23 MB 2025-02-15 05:49:06,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 87.59 MB 2025-02-15 05:49:06,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19046.39 MB 2025-02-15 05:49:06,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:49:06,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:49:06,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:49:06,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15048.16 MB 2025-02-15 05:49:06,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15359.86 MB 2025-02-15 05:49:06,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.70 MB 2025-02-15 05:49:06,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15593.74 MB 2025-02-15 05:49:06,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:49:06,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:49:06,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:49:06,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15359.86 MB 2025-02-15 05:49:06,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15738.72 MB 2025-02-15 05:49:06,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.86 MB 2025-02-15 05:49:06,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.58 MB 2025-02-15 05:49:06,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:49:06,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:49:06,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:49:06,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15048.16 MB 2025-02-15 05:49:06,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15738.72 MB 2025-02-15 05:49:06,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.56 MB 2025-02-15 05:49:06,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19522.39 MB 2025-02-15 05:49:06,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.58 MB 2025-02-15 05:49:06,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:49:06,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:49:06,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:49:06,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16104.29 MB 2025-02-15 05:49:06,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16263.29 MB 2025-02-15 05:49:06,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.00 MB 2025-02-15 05:49:06,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19522.39 MB 2025-02-15 05:49:06,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19620.95 MB 2025-02-15 05:49:06,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 98.57 MB 2025-02-15 05:49:06,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16380.07 MB 2025-02-15 05:49:06,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:49:06,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:49:06,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:49:06,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16363.86 MB 2025-02-15 05:49:06,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16522.57 MB 2025-02-15 05:49:06,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 158.71 MB 2025-02-15 05:49:06,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19620.95 MB 2025-02-15 05:49:06,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19620.95 MB 2025-02-15 05:49:06,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:06,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16522.57 MB 2025-02-15 05:49:06,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:49:06,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:49:06,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.99 seconds 2025-02-15 05:49:06,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:06,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13993.03 MB 2025-02-15 05:49:06,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16664.74 MB 2025-02-15 05:49:06,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2671.71 MB 2025-02-15 05:49:06,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44887.44 MB 2025-02-15 05:49:06,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19620.95 MB 2025-02-15 05:49:06,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25266.49 MB 2025-02-15 05:49:06,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16664.74 MB 2025-02-15 05:49:07,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:49:07,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:49:07,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 05:49:07,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:07,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16664.74 MB 2025-02-15 05:49:07,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16508.79 MB 2025-02-15 05:49:07,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -155.95 MB 2025-02-15 05:49:07,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19620.95 MB 2025-02-15 05:49:07,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19620.95 MB 2025-02-15 05:49:07,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:07,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18653.77 MB 2025-02-15 05:49:07,189 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5767, cut from 5769 2025-02-15 05:49:07,189 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:49:07,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:49:07,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:49:07,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:49:07,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:07,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16508.79 MB 2025-02-15 05:49:07,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22475.47 MB 2025-02-15 05:49:07,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5966.68 MB 2025-02-15 05:49:07,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19620.95 MB 2025-02-15 05:49:07,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27038.58 MB 2025-02-15 05:49:07,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7417.63 MB 2025-02-15 05:49:07,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22475.47 MB 2025-02-15 05:49:07,310 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5559] 2025-02-15 05:49:07,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:07,311 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:49:07,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:07,312 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:49:07,317 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:49:07,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:07,318 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:49:07,318 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:49:30,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:30,377 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:49:30,385 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:49:30,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:30,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1007, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:49:30,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:30,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1007, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:49:45,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:49:45,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:49:45,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.55 seconds 2025-02-15 05:49:45,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:45,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19985.64 MB 2025-02-15 05:49:45,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23549.36 MB 2025-02-15 05:49:45,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3563.72 MB 2025-02-15 05:49:45,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35936.80 MB 2025-02-15 05:49:45,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28074.57 MB 2025-02-15 05:49:45,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7862.22 MB 2025-02-15 05:49:45,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32402.22 MB 2025-02-15 05:49:46,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:49:46,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:49:46,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:49:46,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:46,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23549.36 MB 2025-02-15 05:49:46,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21012.93 MB 2025-02-15 05:49:46,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2536.43 MB 2025-02-15 05:49:46,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28074.57 MB 2025-02-15 05:49:46,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35771.12 MB 2025-02-15 05:49:46,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7696.55 MB 2025-02-15 05:49:46,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34053.67 MB 2025-02-15 05:49:47,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:49:47,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:49:47,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:49:47,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:47,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21012.93 MB 2025-02-15 05:49:47,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21543.90 MB 2025-02-15 05:49:47,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.97 MB 2025-02-15 05:49:47,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35771.12 MB 2025-02-15 05:49:47,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23825.74 MB 2025-02-15 05:49:47,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11945.38 MB 2025-02-15 05:49:47,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25523.36 MB 2025-02-15 05:49:47,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:49:47,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:49:47,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:49:47,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:47,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21543.90 MB 2025-02-15 05:49:47,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23433.44 MB 2025-02-15 05:49:47,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:49:47,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23825.74 MB 2025-02-15 05:49:47,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26656.90 MB 2025-02-15 05:49:47,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:49:47,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24850.87 MB 2025-02-15 05:49:48,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:49:48,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:49:48,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:49:48,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23433.44 MB 2025-02-15 05:49:48,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25676.34 MB 2025-02-15 05:49:48,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 05:49:48,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26656.90 MB 2025-02-15 05:49:48,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33028.05 MB 2025-02-15 05:49:48,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 05:49:48,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31220.62 MB 2025-02-15 05:49:48,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:49:48,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:49:48,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:49:48,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21543.90 MB 2025-02-15 05:49:48,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25676.34 MB 2025-02-15 05:49:48,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 05:49:48,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23825.74 MB 2025-02-15 05:49:48,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33028.05 MB 2025-02-15 05:49:48,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-15 05:49:48,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31220.62 MB 2025-02-15 05:49:48,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:49:48,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:49:48,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:49:48,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27209.88 MB 2025-02-15 05:49:48,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27976.89 MB 2025-02-15 05:49:48,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:49:48,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33028.05 MB 2025-02-15 05:49:48,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33443.28 MB 2025-02-15 05:49:48,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:49:48,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28684.67 MB 2025-02-15 05:49:48,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:49:48,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:49:48,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:49:48,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28389.77 MB 2025-02-15 05:49:48,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.32 MB 2025-02-15 05:49:48,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.55 MB 2025-02-15 05:49:48,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33443.28 MB 2025-02-15 05:49:48,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33443.28 MB 2025-02-15 05:49:48,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:48,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28824.27 MB 2025-02-15 05:49:48,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:49:48,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:49:48,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.98 seconds 2025-02-15 05:49:48,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16477.17 MB 2025-02-15 05:49:48,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28817.18 MB 2025-02-15 05:49:48,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12340.00 MB 2025-02-15 05:49:48,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35936.80 MB 2025-02-15 05:49:48,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33443.28 MB 2025-02-15 05:49:48,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2493.51 MB 2025-02-15 05:49:48,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28824.27 MB 2025-02-15 05:49:48,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:49:48,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:49:48,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:49:48,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28817.18 MB 2025-02-15 05:49:48,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21465.31 MB 2025-02-15 05:49:48,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7351.87 MB 2025-02-15 05:49:48,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33443.28 MB 2025-02-15 05:49:48,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33443.28 MB 2025-02-15 05:49:48,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:49:48,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31315.02 MB 2025-02-15 05:49:48,659 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 05:49:48,659 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:49:48,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:49:48,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:49:48,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:49:48,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:49:48,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21465.31 MB 2025-02-15 05:49:48,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29857.90 MB 2025-02-15 05:49:48,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-15 05:49:48,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33443.28 MB 2025-02-15 05:49:48,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41787.85 MB 2025-02-15 05:49:48,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-15 05:49:48,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29857.90 MB 2025-02-15 05:49:48,823 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 05:49:48,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:48,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:49:48,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:48,826 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:49:48,830 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:49:48,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:49:48,831 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:49:48,831 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:50:51,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:50:51,554 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:50:51,559 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:50:51,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:50:51,562 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 508, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:50:51,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:50:51,563 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 508, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:50:59,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:50:59,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:50:59,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.80 seconds 2025-02-15 05:50:59,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:50:59,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16508.53 MB 2025-02-15 05:50:59,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.32 MB 2025-02-15 05:50:59,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1797.78 MB 2025-02-15 05:50:59,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54303.65 MB 2025-02-15 05:50:59,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20252.20 MB 2025-02-15 05:50:59,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34051.46 MB 2025-02-15 05:50:59,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27112.36 MB 2025-02-15 05:50:59,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:50:59,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:50:59,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:50:59,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:50:59,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18306.32 MB 2025-02-15 05:50:59,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18419.83 MB 2025-02-15 05:50:59,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 113.52 MB 2025-02-15 05:50:59,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20252.20 MB 2025-02-15 05:50:59,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25748.83 MB 2025-02-15 05:50:59,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5496.64 MB 2025-02-15 05:50:59,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25903.20 MB 2025-02-15 05:51:01,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:51:01,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:51:01,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 05:51:01,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18419.83 MB 2025-02-15 05:51:01,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18950.67 MB 2025-02-15 05:51:01,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:51:01,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25748.83 MB 2025-02-15 05:51:01,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 05:51:01,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4315.94 MB 2025-02-15 05:51:01,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22931.30 MB 2025-02-15 05:51:01,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:51:01,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:51:01,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 05:51:01,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18950.67 MB 2025-02-15 05:51:01,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20840.21 MB 2025-02-15 05:51:01,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:51:01,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 05:51:01,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24264.05 MB 2025-02-15 05:51:01,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 05:51:01,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22257.64 MB 2025-02-15 05:51:01,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:51:01,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:51:01,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:51:01,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20840.21 MB 2025-02-15 05:51:01,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23082.06 MB 2025-02-15 05:51:01,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:51:01,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24264.05 MB 2025-02-15 05:51:01,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 05:51:01,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:51:01,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28626.35 MB 2025-02-15 05:51:01,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:51:01,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:51:01,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 05:51:01,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18950.67 MB 2025-02-15 05:51:01,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23082.06 MB 2025-02-15 05:51:01,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:51:01,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 05:51:01,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 05:51:01,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 05:51:01,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28626.35 MB 2025-02-15 05:51:01,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:51:01,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:51:01,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:51:01,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24615.61 MB 2025-02-15 05:51:01,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25382.61 MB 2025-02-15 05:51:01,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:51:01,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30398.22 MB 2025-02-15 05:51:01,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 05:51:01,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:51:01,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26090.40 MB 2025-02-15 05:51:01,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:51:01,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:51:01,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:51:01,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25795.50 MB 2025-02-15 05:51:01,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26022.19 MB 2025-02-15 05:51:01,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.69 MB 2025-02-15 05:51:01,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 05:51:01,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 05:51:01,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:51:01,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26185.96 MB 2025-02-15 05:51:01,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:51:01,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:51:01,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.22 seconds 2025-02-15 05:51:01,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:01,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14738.62 MB 2025-02-15 05:51:01,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26223.26 MB 2025-02-15 05:51:01,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11484.64 MB 2025-02-15 05:51:01,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54303.65 MB 2025-02-15 05:51:01,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 05:51:01,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23488.10 MB 2025-02-15 05:51:01,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26223.26 MB 2025-02-15 05:51:02,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:51:02,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:51:02,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:51:02,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:02,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26223.26 MB 2025-02-15 05:51:02,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19743.01 MB 2025-02-15 05:51:02,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6480.25 MB 2025-02-15 05:51:02,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 05:51:02,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 05:51:02,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:51:02,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28734.93 MB 2025-02-15 05:51:02,074 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:51:02,074 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:51:02,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:51:02,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:51:02,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:51:02,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:51:02,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19743.01 MB 2025-02-15 05:51:02,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28182.03 MB 2025-02-15 05:51:02,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:51:02,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 05:51:02,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 05:51:02,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:51:02,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28182.03 MB 2025-02-15 05:51:02,243 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:51:02,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:02,244 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:51:02,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:02,245 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:51:02,250 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:51:02,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:02,251 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:51:02,251 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:51:44,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:44,244 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:51:44,249 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:51:44,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:44,254 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:51:44,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:51:44,255 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:52:04,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:52:04,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:52:04,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.42 seconds 2025-02-15 05:52:04,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:04,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-15 05:52:04,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.63 MB 2025-02-15 05:52:04,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-15 05:52:04,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 05:52:04,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37956.35 MB 2025-02-15 05:52:04,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15934.16 MB 2025-02-15 05:52:04,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35763.69 MB 2025-02-15 05:52:04,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:52:04,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:52:04,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:52:04,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:04,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.63 MB 2025-02-15 05:52:04,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22676.51 MB 2025-02-15 05:52:04,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4235.12 MB 2025-02-15 05:52:04,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37956.35 MB 2025-02-15 05:52:04,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47229.96 MB 2025-02-15 05:52:04,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9273.61 MB 2025-02-15 05:52:04,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40855.05 MB 2025-02-15 05:52:06,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:52:06,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:52:06,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:52:06,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:06,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22676.51 MB 2025-02-15 05:52:06,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23207.35 MB 2025-02-15 05:52:06,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:52:06,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47229.96 MB 2025-02-15 05:52:06,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:52:06,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18165.53 MB 2025-02-15 05:52:06,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27185.90 MB 2025-02-15 05:52:06,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:52:06,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:52:06,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:52:06,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:06,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-15 05:52:06,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25096.89 MB 2025-02-15 05:52:06,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:52:06,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:52:06,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:52:06,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:06,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26514.31 MB 2025-02-15 05:52:06,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:52:06,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:52:06,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:52:06,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:06,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25096.89 MB 2025-02-15 05:52:06,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-15 05:52:06,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:52:06,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:52:06,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:52:06,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:52:06,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-15 05:52:06,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:52:06,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:52:06,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:52:06,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:06,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-15 05:52:06,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-15 05:52:06,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:52:06,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:52:06,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:52:06,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:52:06,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-15 05:52:07,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:52:07,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:52:07,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 05:52:07,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:07,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28872.28 MB 2025-02-15 05:52:07,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.29 MB 2025-02-15 05:52:07,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:52:07,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34726.74 MB 2025-02-15 05:52:07,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:52:07,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:52:07,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30347.07 MB 2025-02-15 05:52:07,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:52:07,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:52:07,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:52:07,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:07,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30052.17 MB 2025-02-15 05:52:07,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30279.53 MB 2025-02-15 05:52:07,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.35 MB 2025-02-15 05:52:07,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:52:07,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:52:07,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:07,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.90 MB 2025-02-15 05:52:07,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:52:07,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:52:07,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.85 seconds 2025-02-15 05:52:07,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:07,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17592.08 MB 2025-02-15 05:52:07,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30480.38 MB 2025-02-15 05:52:07,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12888.30 MB 2025-02-15 05:52:07,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 05:52:07,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:52:07,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18748.54 MB 2025-02-15 05:52:07,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.90 MB 2025-02-15 05:52:07,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:52:07,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:52:07,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 05:52:07,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:07,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30480.38 MB 2025-02-15 05:52:07,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.83 MB 2025-02-15 05:52:07,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7890.54 MB 2025-02-15 05:52:07,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:52:07,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:52:07,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:07,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32986.51 MB 2025-02-15 05:52:07,415 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 05:52:07,416 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:52:07,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:52:07,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:52:07,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:52:07,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:07,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22589.83 MB 2025-02-15 05:52:07,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31010.61 MB 2025-02-15 05:52:07,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 05:52:07,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:52:07,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39327.89 MB 2025-02-15 05:52:07,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 05:52:07,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31010.61 MB 2025-02-15 05:52:07,658 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 05:52:07,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:07,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:52:07,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:07,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:52:07,670 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:52:07,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:07,672 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:52:07,673 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:52:16,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:16,747 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:52:16,751 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:52:16,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:16,755 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1051, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:52:16,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:16,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1051, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:52:33,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:52:33,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:52:33,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.36 seconds 2025-02-15 05:52:33,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:33,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20292.24 MB 2025-02-15 05:52:33,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24012.59 MB 2025-02-15 05:52:33,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3720.35 MB 2025-02-15 05:52:33,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47699.72 MB 2025-02-15 05:52:33,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28569.50 MB 2025-02-15 05:52:33,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19130.22 MB 2025-02-15 05:52:33,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32934.51 MB 2025-02-15 05:52:33,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:52:33,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:52:33,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:52:33,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:33,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24012.59 MB 2025-02-15 05:52:33,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21242.72 MB 2025-02-15 05:52:33,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2769.87 MB 2025-02-15 05:52:33,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28569.50 MB 2025-02-15 05:52:33,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38077.99 MB 2025-02-15 05:52:33,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9508.49 MB 2025-02-15 05:52:33,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35621.00 MB 2025-02-15 05:52:35,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:52:35,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:52:35,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 05:52:35,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21242.72 MB 2025-02-15 05:52:35,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21773.56 MB 2025-02-15 05:52:35,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:52:35,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38077.99 MB 2025-02-15 05:52:35,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26973.57 MB 2025-02-15 05:52:35,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11104.42 MB 2025-02-15 05:52:35,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25752.11 MB 2025-02-15 05:52:35,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:52:35,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:52:35,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:52:35,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21773.56 MB 2025-02-15 05:52:35,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23663.10 MB 2025-02-15 05:52:35,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:52:35,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26973.57 MB 2025-02-15 05:52:35,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-15 05:52:35,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 05:52:35,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25080.52 MB 2025-02-15 05:52:35,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:52:35,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:52:35,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:52:35,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23663.10 MB 2025-02-15 05:52:35,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25904.95 MB 2025-02-15 05:52:35,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:52:35,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-15 05:52:35,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33579.60 MB 2025-02-15 05:52:35,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:52:35,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31449.23 MB 2025-02-15 05:52:35,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:52:35,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:52:35,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:52:35,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21773.56 MB 2025-02-15 05:52:35,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25904.95 MB 2025-02-15 05:52:35,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:52:35,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26973.57 MB 2025-02-15 05:52:35,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33579.60 MB 2025-02-15 05:52:35,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 05:52:35,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31449.23 MB 2025-02-15 05:52:35,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:52:35,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:52:35,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:52:35,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27438.49 MB 2025-02-15 05:52:35,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28205.50 MB 2025-02-15 05:52:35,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:52:35,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33579.60 MB 2025-02-15 05:52:35,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-15 05:52:35,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:52:35,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28913.28 MB 2025-02-15 05:52:35,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:52:35,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:52:35,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:52:35,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28618.38 MB 2025-02-15 05:52:35,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28846.79 MB 2025-02-15 05:52:35,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.41 MB 2025-02-15 05:52:35,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33996.93 MB 2025-02-15 05:52:35,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-15 05:52:35,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:35,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29080.15 MB 2025-02-15 05:52:35,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:52:35,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:52:35,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.78 seconds 2025-02-15 05:52:35,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16630.47 MB 2025-02-15 05:52:35,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29047.64 MB 2025-02-15 05:52:35,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12417.17 MB 2025-02-15 05:52:35,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47699.72 MB 2025-02-15 05:52:35,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-15 05:52:35,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13702.79 MB 2025-02-15 05:52:35,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29080.15 MB 2025-02-15 05:52:35,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:52:35,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:52:35,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:52:35,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29047.64 MB 2025-02-15 05:52:35,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21620.39 MB 2025-02-15 05:52:35,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7427.26 MB 2025-02-15 05:52:35,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33996.93 MB 2025-02-15 05:52:35,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-15 05:52:35,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:35,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31547.02 MB 2025-02-15 05:52:35,828 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 05:52:35,828 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:52:35,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:52:35,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:52:35,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:52:35,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:35,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21620.39 MB 2025-02-15 05:52:35,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30017.79 MB 2025-02-15 05:52:35,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 05:52:35,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33996.93 MB 2025-02-15 05:52:35,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42347.79 MB 2025-02-15 05:52:35,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 05:52:35,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30017.79 MB 2025-02-15 05:52:35,991 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 05:52:35,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:35,993 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:52:35,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:35,994 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:52:35,998 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:52:35,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:35,999 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:52:35,999 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:52:46,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:46,179 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:52:46,184 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:52:46,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:46,187 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:52:46,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:46,188 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:52:49,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:52:49,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:52:49,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-15 05:52:49,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-15 05:52:49,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-15 05:52:49,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-15 05:52:49,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-15 05:52:49,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22317.89 MB 2025-02-15 05:52:49,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28380.76 MB 2025-02-15 05:52:49,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-15 05:52:49,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:52:49,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:52:49,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:52:49,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-15 05:52:49,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15061.45 MB 2025-02-15 05:52:49,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.95 MB 2025-02-15 05:52:49,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22317.89 MB 2025-02-15 05:52:49,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22317.89 MB 2025-02-15 05:52:49,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:49,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17174.11 MB 2025-02-15 05:52:49,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:52:49,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:52:49,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 05:52:49,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15061.45 MB 2025-02-15 05:52:49,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15279.09 MB 2025-02-15 05:52:49,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-15 05:52:49,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22317.89 MB 2025-02-15 05:52:49,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21846.03 MB 2025-02-15 05:52:49,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 05:52:49,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19231.90 MB 2025-02-15 05:52:49,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:52:49,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:52:49,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:52:49,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.02 MB 2025-02-15 05:52:49,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16053.55 MB 2025-02-15 05:52:49,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-15 05:52:49,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21846.03 MB 2025-02-15 05:52:49,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21846.03 MB 2025-02-15 05:52:49,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:49,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16634.70 MB 2025-02-15 05:52:49,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:52:49,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:52:49,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:52:49,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16053.55 MB 2025-02-15 05:52:49,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16972.74 MB 2025-02-15 05:52:49,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.20 MB 2025-02-15 05:52:49,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21846.03 MB 2025-02-15 05:52:49,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21846.03 MB 2025-02-15 05:52:49,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:49,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19245.86 MB 2025-02-15 05:52:49,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:52:49,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:52:49,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:52:49,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:49,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.02 MB 2025-02-15 05:52:49,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16972.74 MB 2025-02-15 05:52:49,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.72 MB 2025-02-15 05:52:49,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21846.03 MB 2025-02-15 05:52:49,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21846.03 MB 2025-02-15 05:52:49,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:49,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19245.86 MB 2025-02-15 05:52:50,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:52:50,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:52:50,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:52:50,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:50,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17601.50 MB 2025-02-15 05:52:50,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17915.97 MB 2025-02-15 05:52:50,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-15 05:52:50,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21846.03 MB 2025-02-15 05:52:50,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22013.80 MB 2025-02-15 05:52:50,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 05:52:50,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18214.25 MB 2025-02-15 05:52:50,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:52:50,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:52:50,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:52:50,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:50,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18085.26 MB 2025-02-15 05:52:50,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18298.95 MB 2025-02-15 05:52:50,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.69 MB 2025-02-15 05:52:50,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22013.80 MB 2025-02-15 05:52:50,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22013.80 MB 2025-02-15 05:52:50,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:50,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18325.90 MB 2025-02-15 05:52:50,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:52:50,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:52:50,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.84 seconds 2025-02-15 05:52:50,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:50,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-15 05:52:50,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18499.97 MB 2025-02-15 05:52:50,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4900.65 MB 2025-02-15 05:52:50,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-15 05:52:50,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22013.80 MB 2025-02-15 05:52:50,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28684.85 MB 2025-02-15 05:52:50,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18499.97 MB 2025-02-15 05:52:50,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:52:50,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:52:50,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:52:50,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:50,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18499.97 MB 2025-02-15 05:52:50,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17489.20 MB 2025-02-15 05:52:50,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1010.77 MB 2025-02-15 05:52:50,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22013.80 MB 2025-02-15 05:52:50,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22013.80 MB 2025-02-15 05:52:50,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:52:50,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19203.06 MB 2025-02-15 05:52:50,316 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 05:52:50,317 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:52:50,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:52:50,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:52:50,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:52:50,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:52:50,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17489.20 MB 2025-02-15 05:52:50,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25926.67 MB 2025-02-15 05:52:50,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 05:52:50,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22013.80 MB 2025-02-15 05:52:50,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30402.41 MB 2025-02-15 05:52:50,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 05:52:50,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25926.67 MB 2025-02-15 05:52:50,564 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 05:52:50,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:50,567 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:52:50,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:50,568 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:52:50,576 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:52:50,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:52:50,578 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:52:50,578 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:53:19,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:19,349 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:53:19,356 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:53:19,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:19,363 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:53:19,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:19,365 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:53:22,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:53:22,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:53:22,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-15 05:53:22,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:22,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-15 05:53:22,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-15 05:53:22,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-15 05:53:22,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38791.02 MB 2025-02-15 05:53:22,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19014.88 MB 2025-02-15 05:53:22,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19776.14 MB 2025-02-15 05:53:22,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-15 05:53:22,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:53:22,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:53:22,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:53:22,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:22,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-15 05:53:22,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15089.54 MB 2025-02-15 05:53:22,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.04 MB 2025-02-15 05:53:22,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19014.88 MB 2025-02-15 05:53:22,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19014.88 MB 2025-02-15 05:53:22,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:53:22,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17230.29 MB 2025-02-15 05:53:23,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:53:23,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:53:23,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 05:53:23,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15089.54 MB 2025-02-15 05:53:23,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15312.49 MB 2025-02-15 05:53:23,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.95 MB 2025-02-15 05:53:23,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19014.88 MB 2025-02-15 05:53:23,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17437.82 MB 2025-02-15 05:53:23,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1577.06 MB 2025-02-15 05:53:23,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19260.23 MB 2025-02-15 05:53:23,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:53:23,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:53:23,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:53:23,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.42 MB 2025-02-15 05:53:23,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16105.84 MB 2025-02-15 05:53:23,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 793.41 MB 2025-02-15 05:53:23,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17437.82 MB 2025-02-15 05:53:23,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-15 05:53:23,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 396.36 MB 2025-02-15 05:53:23,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16701.16 MB 2025-02-15 05:53:23,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:53:23,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:53:23,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 05:53:23,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16105.84 MB 2025-02-15 05:53:23,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.45 MB 2025-02-15 05:53:23,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 941.62 MB 2025-02-15 05:53:23,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-15 05:53:23,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20411.58 MB 2025-02-15 05:53:23,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2577.40 MB 2025-02-15 05:53:23,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19377.06 MB 2025-02-15 05:53:23,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:53:23,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:53:23,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 05:53:23,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.42 MB 2025-02-15 05:53:23,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.45 MB 2025-02-15 05:53:23,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1735.03 MB 2025-02-15 05:53:23,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17437.82 MB 2025-02-15 05:53:23,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20411.58 MB 2025-02-15 05:53:23,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2973.76 MB 2025-02-15 05:53:23,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19377.06 MB 2025-02-15 05:53:23,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:53:23,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:53:23,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:53:23,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17692.59 MB 2025-02-15 05:53:23,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18014.73 MB 2025-02-15 05:53:23,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.14 MB 2025-02-15 05:53:23,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20411.58 MB 2025-02-15 05:53:23,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-15 05:53:23,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 05:53:23,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18318.92 MB 2025-02-15 05:53:23,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:53:23,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:53:23,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:53:23,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18188.15 MB 2025-02-15 05:53:23,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18415.51 MB 2025-02-15 05:53:23,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.36 MB 2025-02-15 05:53:23,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-15 05:53:23,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-15 05:53:23,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:53:23,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18443.27 MB 2025-02-15 05:53:23,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:53:23,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:53:23,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.99 seconds 2025-02-15 05:53:23,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-15 05:53:23,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18616.56 MB 2025-02-15 05:53:23,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5017.23 MB 2025-02-15 05:53:23,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38791.02 MB 2025-02-15 05:53:23,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-15 05:53:23,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18205.38 MB 2025-02-15 05:53:23,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18616.56 MB 2025-02-15 05:53:23,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:53:23,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:53:23,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 05:53:23,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18616.56 MB 2025-02-15 05:53:23,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17509.51 MB 2025-02-15 05:53:23,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1107.05 MB 2025-02-15 05:53:23,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-15 05:53:23,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-15 05:53:23,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:53:23,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19219.28 MB 2025-02-15 05:53:23,665 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 05:53:23,666 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:53:23,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:53:23,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:53:23,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:53:23,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:53:23,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17509.51 MB 2025-02-15 05:53:23,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25948.34 MB 2025-02-15 05:53:23,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 05:53:23,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-15 05:53:23,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31071.40 MB 2025-02-15 05:53:23,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 05:53:23,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25948.34 MB 2025-02-15 05:53:23,927 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 05:53:23,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:23,930 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:53:23,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:23,932 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:53:23,939 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:53:23,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:53:23,941 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:53:23,942 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:54:09,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:09,804 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:54:09,812 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:54:09,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:09,819 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 637, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:54:09,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:09,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 637, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:54:19,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:54:19,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:54:19,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.90 seconds 2025-02-15 05:54:19,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:19,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17407.42 MB 2025-02-15 05:54:19,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19661.86 MB 2025-02-15 05:54:19,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2254.44 MB 2025-02-15 05:54:19,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39460.01 MB 2025-02-15 05:54:19,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25023.22 MB 2025-02-15 05:54:19,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14436.79 MB 2025-02-15 05:54:19,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28464.24 MB 2025-02-15 05:54:19,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:54:19,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:54:19,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 05:54:19,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:19,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19661.86 MB 2025-02-15 05:54:19,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19090.46 MB 2025-02-15 05:54:19,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -571.40 MB 2025-02-15 05:54:19,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25023.22 MB 2025-02-15 05:54:19,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31539.07 MB 2025-02-15 05:54:19,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6515.85 MB 2025-02-15 05:54:19,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28277.76 MB 2025-02-15 05:54:21,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:54:21,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:54:21,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:54:21,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:21,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19090.46 MB 2025-02-15 05:54:21,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19621.31 MB 2025-02-15 05:54:21,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:54:21,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31539.07 MB 2025-02-15 05:54:21,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24893.19 MB 2025-02-15 05:54:21,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6645.87 MB 2025-02-15 05:54:21,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23599.85 MB 2025-02-15 05:54:21,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:54:21,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:54:21,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:54:21,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:21,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.31 MB 2025-02-15 05:54:21,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21510.84 MB 2025-02-15 05:54:21,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:54:21,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 05:54:21,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24893.19 MB 2025-02-15 05:54:21,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:54:21,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22928.27 MB 2025-02-15 05:54:21,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:54:21,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:54:21,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:54:21,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:21,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21510.84 MB 2025-02-15 05:54:21,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23752.70 MB 2025-02-15 05:54:21,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:54:21,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 05:54:21,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31501.32 MB 2025-02-15 05:54:21,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 05:54:21,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29298.03 MB 2025-02-15 05:54:21,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:54:21,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:54:21,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:54:21,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:21,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.31 MB 2025-02-15 05:54:21,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23752.70 MB 2025-02-15 05:54:21,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:54:21,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 05:54:21,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31501.32 MB 2025-02-15 05:54:21,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 05:54:21,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29298.03 MB 2025-02-15 05:54:22,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:54:22,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:54:22,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:54:22,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:22,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25287.29 MB 2025-02-15 05:54:22,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26054.29 MB 2025-02-15 05:54:22,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:54:22,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31501.32 MB 2025-02-15 05:54:22,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 05:54:22,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:54:22,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26762.08 MB 2025-02-15 05:54:22,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:54:22,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:54:22,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:54:22,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:22,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26467.18 MB 2025-02-15 05:54:22,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26697.15 MB 2025-02-15 05:54:22,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.97 MB 2025-02-15 05:54:22,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 05:54:22,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 05:54:22,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:54:22,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26918.67 MB 2025-02-15 05:54:22,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:54:22,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:54:22,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.27 seconds 2025-02-15 05:54:22,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:22,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15188.07 MB 2025-02-15 05:54:22,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26898.22 MB 2025-02-15 05:54:22,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11710.15 MB 2025-02-15 05:54:22,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39460.01 MB 2025-02-15 05:54:22,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 05:54:22,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7541.36 MB 2025-02-15 05:54:22,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26918.67 MB 2025-02-15 05:54:22,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:54:22,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:54:22,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:54:22,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:22,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26898.22 MB 2025-02-15 05:54:22,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20193.50 MB 2025-02-15 05:54:22,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6704.72 MB 2025-02-15 05:54:22,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 05:54:22,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 05:54:22,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:54:22,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29409.89 MB 2025-02-15 05:54:22,384 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:54:22,385 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:54:22,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:54:22,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:54:22,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:54:22,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:22,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20193.50 MB 2025-02-15 05:54:22,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28632.53 MB 2025-02-15 05:54:22,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:54:22,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 05:54:22,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36115.05 MB 2025-02-15 05:54:22,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 05:54:22,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28632.53 MB 2025-02-15 05:54:22,553 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:54:22,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:22,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:54:22,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:22,556 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:54:22,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:54:22,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:22,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:54:22,562 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:54:32,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:32,360 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:54:32,364 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:54:32,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:32,368 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1003, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:54:32,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:32,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1003, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:54:47,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:54:47,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:54:47,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.59 seconds 2025-02-15 05:54:47,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:47,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19957.77 MB 2025-02-15 05:54:47,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.25 MB 2025-02-15 05:54:47,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3550.48 MB 2025-02-15 05:54:47,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48700.06 MB 2025-02-15 05:54:47,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26321.35 MB 2025-02-15 05:54:47,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22378.71 MB 2025-02-15 05:54:47,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32374.35 MB 2025-02-15 05:54:48,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:54:48,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:54:48,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:54:48,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:48,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.25 MB 2025-02-15 05:54:48,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20993.18 MB 2025-02-15 05:54:48,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2515.07 MB 2025-02-15 05:54:48,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26321.35 MB 2025-02-15 05:54:48,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35204.89 MB 2025-02-15 05:54:48,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8883.54 MB 2025-02-15 05:54:48,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34491.89 MB 2025-02-15 05:54:50,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:54:50,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:54:50,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 05:54:50,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20993.18 MB 2025-02-15 05:54:50,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21524.03 MB 2025-02-15 05:54:50,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:54:50,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35204.89 MB 2025-02-15 05:54:50,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22798.14 MB 2025-02-15 05:54:50,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12406.75 MB 2025-02-15 05:54:50,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25504.65 MB 2025-02-15 05:54:50,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:54:50,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:54:50,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:54:50,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21524.03 MB 2025-02-15 05:54:50,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23413.56 MB 2025-02-15 05:54:50,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:54:50,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22798.14 MB 2025-02-15 05:54:50,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26573.01 MB 2025-02-15 05:54:50,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:54:50,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24830.99 MB 2025-02-15 05:54:50,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:54:50,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:54:50,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:54:50,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23413.56 MB 2025-02-15 05:54:50,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25655.42 MB 2025-02-15 05:54:50,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:54:50,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26573.01 MB 2025-02-15 05:54:50,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32707.18 MB 2025-02-15 05:54:50,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 05:54:50,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.70 MB 2025-02-15 05:54:50,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:54:50,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:54:50,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:54:50,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21524.03 MB 2025-02-15 05:54:50,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25655.42 MB 2025-02-15 05:54:50,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:54:50,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22798.14 MB 2025-02-15 05:54:50,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32707.18 MB 2025-02-15 05:54:50,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 05:54:50,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.70 MB 2025-02-15 05:54:50,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:54:50,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:54:50,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:54:50,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27188.96 MB 2025-02-15 05:54:50,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27955.96 MB 2025-02-15 05:54:50,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:54:50,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32707.18 MB 2025-02-15 05:54:50,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33122.42 MB 2025-02-15 05:54:50,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:54:50,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28663.75 MB 2025-02-15 05:54:50,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:54:50,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:54:50,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:54:50,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28368.85 MB 2025-02-15 05:54:50,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28597.06 MB 2025-02-15 05:54:50,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.21 MB 2025-02-15 05:54:50,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33122.42 MB 2025-02-15 05:54:50,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33122.42 MB 2025-02-15 05:54:50,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:54:50,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28780.04 MB 2025-02-15 05:54:50,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:54:50,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:54:50,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.04 seconds 2025-02-15 05:54:50,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16463.24 MB 2025-02-15 05:54:50,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28797.91 MB 2025-02-15 05:54:50,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12334.67 MB 2025-02-15 05:54:50,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48700.06 MB 2025-02-15 05:54:50,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33122.42 MB 2025-02-15 05:54:50,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15577.65 MB 2025-02-15 05:54:50,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28797.91 MB 2025-02-15 05:54:50,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:54:50,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:54:50,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:54:50,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28797.91 MB 2025-02-15 05:54:50,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21462.42 MB 2025-02-15 05:54:50,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7335.49 MB 2025-02-15 05:54:50,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33122.42 MB 2025-02-15 05:54:50,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33122.42 MB 2025-02-15 05:54:50,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:54:50,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31305.28 MB 2025-02-15 05:54:50,703 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 05:54:50,703 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:54:50,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:54:50,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:54:50,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:54:50,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:54:50,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21462.42 MB 2025-02-15 05:54:50,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29887.37 MB 2025-02-15 05:54:50,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 05:54:50,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33122.42 MB 2025-02-15 05:54:50,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41498.44 MB 2025-02-15 05:54:50,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 05:54:50,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29887.37 MB 2025-02-15 05:54:50,868 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 05:54:50,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:50,870 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:54:50,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:50,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:54:50,875 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:54:50,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:50,876 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:54:50,877 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:54:57,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:57,887 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:54:57,895 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:54:57,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:57,901 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:54:57,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:54:57,903 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:55:01,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:55:01,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:55:01,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.10 seconds 2025-02-15 05:55:01,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14299.63 MB 2025-02-15 05:55:01,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.56 MB 2025-02-15 05:55:01,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-15 05:55:01,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49874.47 MB 2025-02-15 05:55:01,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20220.74 MB 2025-02-15 05:55:01,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29653.73 MB 2025-02-15 05:55:01,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23853.72 MB 2025-02-15 05:55:01,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:55:01,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:55:01,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:55:01,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14975.56 MB 2025-02-15 05:55:01,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15204.73 MB 2025-02-15 05:55:01,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-15 05:55:01,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20220.74 MB 2025-02-15 05:55:01,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20220.74 MB 2025-02-15 05:55:01,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:01,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17461.78 MB 2025-02-15 05:55:01,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:55:01,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:55:01,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 05:55:01,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15204.73 MB 2025-02-15 05:55:01,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15439.63 MB 2025-02-15 05:55:01,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 05:55:01,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20220.74 MB 2025-02-15 05:55:01,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19748.88 MB 2025-02-15 05:55:01,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 05:55:01,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19375.42 MB 2025-02-15 05:55:01,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:55:01,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:55:01,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:55:01,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15439.56 MB 2025-02-15 05:55:01,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16275.48 MB 2025-02-15 05:55:01,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 05:55:01,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19748.88 MB 2025-02-15 05:55:01,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19748.88 MB 2025-02-15 05:55:01,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:01,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16902.70 MB 2025-02-15 05:55:01,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:55:01,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:55:01,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:55:01,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16275.48 MB 2025-02-15 05:55:01,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17267.54 MB 2025-02-15 05:55:01,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 05:55:01,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19748.88 MB 2025-02-15 05:55:01,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21007.17 MB 2025-02-15 05:55:01,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-15 05:55:01,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19722.68 MB 2025-02-15 05:55:01,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:55:01,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:55:01,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:55:01,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:01,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15439.56 MB 2025-02-15 05:55:01,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17267.54 MB 2025-02-15 05:55:01,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 05:55:01,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19748.88 MB 2025-02-15 05:55:01,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21007.17 MB 2025-02-15 05:55:01,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-15 05:55:01,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19722.68 MB 2025-02-15 05:55:02,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:55:02,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:55:02,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 05:55:02,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:02,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17946.13 MB 2025-02-15 05:55:02,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18287.36 MB 2025-02-15 05:55:02,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.23 MB 2025-02-15 05:55:02,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21007.17 MB 2025-02-15 05:55:02,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21183.33 MB 2025-02-15 05:55:02,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 176.16 MB 2025-02-15 05:55:02,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18605.63 MB 2025-02-15 05:55:02,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:55:02,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:55:02,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:55:02,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:02,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18470.07 MB 2025-02-15 05:55:02,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18697.54 MB 2025-02-15 05:55:02,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.47 MB 2025-02-15 05:55:02,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21183.33 MB 2025-02-15 05:55:02,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21183.33 MB 2025-02-15 05:55:02,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:02,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18722.02 MB 2025-02-15 05:55:02,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:55:02,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:55:02,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.16 seconds 2025-02-15 05:55:02,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:02,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13634.17 MB 2025-02-15 05:55:02,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18898.62 MB 2025-02-15 05:55:02,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5264.45 MB 2025-02-15 05:55:02,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49874.47 MB 2025-02-15 05:55:02,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21183.33 MB 2025-02-15 05:55:02,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28691.14 MB 2025-02-15 05:55:02,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18898.62 MB 2025-02-15 05:55:02,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:55:02,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:55:02,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:55:02,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:02,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18898.62 MB 2025-02-15 05:55:02,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17587.99 MB 2025-02-15 05:55:02,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1310.63 MB 2025-02-15 05:55:02,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21183.33 MB 2025-02-15 05:55:02,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21183.33 MB 2025-02-15 05:55:02,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:02,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19133.70 MB 2025-02-15 05:55:02,352 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:55:02,352 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:55:02,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:55:02,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:55:02,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:55:02,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:02,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17587.99 MB 2025-02-15 05:55:02,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26027.01 MB 2025-02-15 05:55:02,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:55:02,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21183.33 MB 2025-02-15 05:55:02,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29574.04 MB 2025-02-15 05:55:02,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 05:55:02,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26027.01 MB 2025-02-15 05:55:02,522 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:55:02,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:02,524 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:55:02,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:02,525 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:55:02,529 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:55:02,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:02,531 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:55:02,531 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:55:57,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:57,442 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:55:57,447 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:55:57,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:57,450 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 106, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:55:57,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:55:57,451 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 106, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:55:59,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:55:59,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:55:59,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.63 seconds 2025-02-15 05:55:59,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-15 05:55:59,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.46 MB 2025-02-15 05:55:59,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 375.13 MB 2025-02-15 05:55:59,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42159.05 MB 2025-02-15 05:55:59,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17106.47 MB 2025-02-15 05:55:59,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25052.58 MB 2025-02-15 05:55:59,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22952.21 MB 2025-02-15 05:55:59,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:55:59,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:55:59,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:55:59,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.46 MB 2025-02-15 05:55:59,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14264.21 MB 2025-02-15 05:55:59,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 181.75 MB 2025-02-15 05:55:59,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17106.47 MB 2025-02-15 05:55:59,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17106.47 MB 2025-02-15 05:55:59,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:59,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14826.96 MB 2025-02-15 05:55:59,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:55:59,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:55:59,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.51 seconds 2025-02-15 05:55:59,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.21 MB 2025-02-15 05:55:59,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14404.88 MB 2025-02-15 05:55:59,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 140.67 MB 2025-02-15 05:55:59,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17106.47 MB 2025-02-15 05:55:59,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17106.47 MB 2025-02-15 05:55:59,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:59,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18349.96 MB 2025-02-15 05:55:59,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:55:59,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:55:59,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:55:59,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-15 05:55:59,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14905.42 MB 2025-02-15 05:55:59,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 500.60 MB 2025-02-15 05:55:59,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17106.47 MB 2025-02-15 05:55:59,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17106.47 MB 2025-02-15 05:55:59,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:59,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15281.05 MB 2025-02-15 05:55:59,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:55:59,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:55:59,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 05:55:59,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14905.42 MB 2025-02-15 05:55:59,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-15 05:55:59,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.03 MB 2025-02-15 05:55:59,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17106.47 MB 2025-02-15 05:55:59,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17735.61 MB 2025-02-15 05:55:59,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 629.15 MB 2025-02-15 05:55:59,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16971.11 MB 2025-02-15 05:55:59,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:55:59,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:55:59,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 05:55:59,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-15 05:55:59,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-15 05:55:59,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1108.64 MB 2025-02-15 05:55:59,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17106.47 MB 2025-02-15 05:55:59,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17735.61 MB 2025-02-15 05:55:59,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 629.15 MB 2025-02-15 05:55:59,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16971.11 MB 2025-02-15 05:55:59,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:55:59,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:55:59,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 05:55:59,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16101.43 MB 2025-02-15 05:55:59,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16357.57 MB 2025-02-15 05:55:59,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.14 MB 2025-02-15 05:55:59,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17735.61 MB 2025-02-15 05:55:59,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17899.19 MB 2025-02-15 05:55:59,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 05:55:59,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16545.13 MB 2025-02-15 05:55:59,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:55:59,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:55:59,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:55:59,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16519.09 MB 2025-02-15 05:55:59,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16747.37 MB 2025-02-15 05:55:59,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-15 05:55:59,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17899.19 MB 2025-02-15 05:55:59,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17899.19 MB 2025-02-15 05:55:59,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:55:59,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16747.37 MB 2025-02-15 05:55:59,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:55:59,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:55:59,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.32 seconds 2025-02-15 05:55:59,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:55:59,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13338.02 MB 2025-02-15 05:55:59,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16948.37 MB 2025-02-15 05:55:59,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3610.35 MB 2025-02-15 05:55:59,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42159.05 MB 2025-02-15 05:55:59,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17899.19 MB 2025-02-15 05:55:59,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24259.85 MB 2025-02-15 05:55:59,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16948.37 MB 2025-02-15 05:56:00,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:56:00,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:56:00,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 05:56:00,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:56:00,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14039.86 MB 2025-02-15 05:56:00,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17052.79 MB 2025-02-15 05:56:00,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-15 05:56:00,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17899.19 MB 2025-02-15 05:56:00,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18570.28 MB 2025-02-15 05:56:00,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 671.09 MB 2025-02-15 05:56:00,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17354.35 MB 2025-02-15 05:56:00,056 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 05:56:00,057 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:56:00,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:56:00,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:56:00,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:56:00,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:56:00,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17052.79 MB 2025-02-15 05:56:00,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25488.38 MB 2025-02-15 05:56:00,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 05:56:00,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18570.28 MB 2025-02-15 05:56:00,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 05:56:00,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 05:56:00,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25488.38 MB 2025-02-15 05:56:00,223 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 05:56:00,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:56:00,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:56:00,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:56:00,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:56:00,230 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:56:00,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:56:00,231 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:56:00,231 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 05:57:01,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:01,452 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:57:01,457 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:57:01,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:01,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:57:01,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:01,463 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:57:20,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:57:20,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:57:20,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.63 seconds 2025-02-15 05:57:20,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:20,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-15 05:57:20,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-15 05:57:20,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 05:57:20,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-15 05:57:20,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31270.63 MB 2025-02-15 05:57:20,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6174.02 MB 2025-02-15 05:57:20,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-15 05:57:20,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:57:20,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:57:20,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:57:20,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:20,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-15 05:57:20,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-15 05:57:20,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-15 05:57:20,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31270.63 MB 2025-02-15 05:57:20,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41236.30 MB 2025-02-15 05:57:20,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9965.67 MB 2025-02-15 05:57:20,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38612.19 MB 2025-02-15 05:57:22,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:57:22,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:57:22,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 05:57:22,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-15 05:57:22,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-15 05:57:22,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:57:22,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41236.30 MB 2025-02-15 05:57:22,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24184.36 MB 2025-02-15 05:57:22,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17051.94 MB 2025-02-15 05:57:22,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26615.08 MB 2025-02-15 05:57:22,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:57:22,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:57:22,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:57:22,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 05:57:22,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-15 05:57:22,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:57:22,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24184.36 MB 2025-02-15 05:57:22,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27487.37 MB 2025-02-15 05:57:22,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 05:57:22,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-15 05:57:22,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:57:22,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:57:22,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:57:22,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-15 05:57:22,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 05:57:22,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:57:22,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27487.37 MB 2025-02-15 05:57:22,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34093.40 MB 2025-02-15 05:57:22,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 05:57:22,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-15 05:57:22,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:57:22,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:57:22,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:57:22,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 05:57:22,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 05:57:22,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:57:22,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24184.36 MB 2025-02-15 05:57:22,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34093.40 MB 2025-02-15 05:57:22,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 05:57:22,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-15 05:57:22,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:57:22,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:57:22,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:57:22,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.43 MB 2025-02-15 05:57:22,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.43 MB 2025-02-15 05:57:22,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:57:22,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34093.40 MB 2025-02-15 05:57:22,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34510.73 MB 2025-02-15 05:57:22,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:57:22,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.22 MB 2025-02-15 05:57:22,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:57:22,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:57:22,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:57:22,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.32 MB 2025-02-15 05:57:22,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29707.34 MB 2025-02-15 05:57:22,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.02 MB 2025-02-15 05:57:22,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34510.73 MB 2025-02-15 05:57:22,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34510.73 MB 2025-02-15 05:57:22,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:57:22,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29931.92 MB 2025-02-15 05:57:22,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:57:22,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:57:22,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.12 seconds 2025-02-15 05:57:22,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-15 05:57:22,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29908.19 MB 2025-02-15 05:57:22,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12699.36 MB 2025-02-15 05:57:22,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-15 05:57:22,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34510.73 MB 2025-02-15 05:57:22,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2933.92 MB 2025-02-15 05:57:22,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29931.92 MB 2025-02-15 05:57:22,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:57:22,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:57:22,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:57:22,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29908.19 MB 2025-02-15 05:57:22,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22200.17 MB 2025-02-15 05:57:22,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7708.02 MB 2025-02-15 05:57:22,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34510.73 MB 2025-02-15 05:57:22,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34510.73 MB 2025-02-15 05:57:22,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:57:22,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32408.80 MB 2025-02-15 05:57:22,873 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-15 05:57:22,874 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 05:57:22,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:57:22,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:57:22,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:57:22,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:22,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22200.17 MB 2025-02-15 05:57:22,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30601.70 MB 2025-02-15 05:57:22,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-15 05:57:22,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34510.73 MB 2025-02-15 05:57:22,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42865.79 MB 2025-02-15 05:57:22,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 05:57:22,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30601.70 MB 2025-02-15 05:57:23,041 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-15 05:57:23,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:23,042 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:57:23,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:23,043 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:57:23,048 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:57:23,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:23,049 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:57:23,049 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 05:57:31,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:31,245 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:57:31,250 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:57:31,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:31,253 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1150, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:57:31,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:31,254 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1150, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:57:49,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:57:49,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:57:49,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.87 seconds 2025-02-15 05:57:49,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:49,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.09 MB 2025-02-15 05:57:49,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25052.66 MB 2025-02-15 05:57:49,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4070.57 MB 2025-02-15 05:57:49,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51220.84 MB 2025-02-15 05:57:49,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26822.57 MB 2025-02-15 05:57:49,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24398.27 MB 2025-02-15 05:57:49,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33851.65 MB 2025-02-15 05:57:49,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:57:49,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:57:49,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 05:57:49,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:49,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25052.66 MB 2025-02-15 05:57:49,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21757.39 MB 2025-02-15 05:57:49,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3295.27 MB 2025-02-15 05:57:49,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26822.57 MB 2025-02-15 05:57:49,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36901.49 MB 2025-02-15 05:57:49,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10078.91 MB 2025-02-15 05:57:49,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37229.59 MB 2025-02-15 05:57:51,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:57:51,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:57:51,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 05:57:51,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21757.39 MB 2025-02-15 05:57:51,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22288.23 MB 2025-02-15 05:57:51,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:57:51,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36901.49 MB 2025-02-15 05:57:51,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23311.94 MB 2025-02-15 05:57:51,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13589.54 MB 2025-02-15 05:57:51,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26268.86 MB 2025-02-15 05:57:51,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:57:51,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:57:51,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:57:51,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-15 05:57:51,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24177.50 MB 2025-02-15 05:57:51,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 05:57:51,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23311.94 MB 2025-02-15 05:57:51,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27086.82 MB 2025-02-15 05:57:51,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 05:57:51,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25594.93 MB 2025-02-15 05:57:51,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:57:51,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:57:51,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:57:51,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24177.50 MB 2025-02-15 05:57:51,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.36 MB 2025-02-15 05:57:51,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:57:51,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27086.82 MB 2025-02-15 05:57:51,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33692.84 MB 2025-02-15 05:57:51,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 05:57:51,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.64 MB 2025-02-15 05:57:51,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:57:51,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:57:51,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 05:57:51,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-15 05:57:51,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.36 MB 2025-02-15 05:57:51,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 05:57:51,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23311.94 MB 2025-02-15 05:57:51,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33692.84 MB 2025-02-15 05:57:51,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10380.90 MB 2025-02-15 05:57:51,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.64 MB 2025-02-15 05:57:51,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:57:51,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:57:51,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 05:57:51,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27952.90 MB 2025-02-15 05:57:51,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28719.90 MB 2025-02-15 05:57:51,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:57:51,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33692.84 MB 2025-02-15 05:57:51,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-15 05:57:51,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 05:57:51,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29427.69 MB 2025-02-15 05:57:51,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:57:51,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:57:51,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:57:51,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29132.79 MB 2025-02-15 05:57:51,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.30 MB 2025-02-15 05:57:51,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.51 MB 2025-02-15 05:57:51,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34110.18 MB 2025-02-15 05:57:51,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-15 05:57:51,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:57:51,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29562.53 MB 2025-02-15 05:57:51,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:57:51,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:57:51,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.37 seconds 2025-02-15 05:57:51,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16975.40 MB 2025-02-15 05:57:51,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29562.15 MB 2025-02-15 05:57:51,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12586.75 MB 2025-02-15 05:57:51,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51220.84 MB 2025-02-15 05:57:51,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-15 05:57:51,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17110.66 MB 2025-02-15 05:57:51,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29562.53 MB 2025-02-15 05:57:51,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:57:51,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:57:51,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:57:51,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29562.15 MB 2025-02-15 05:57:51,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21965.67 MB 2025-02-15 05:57:51,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.48 MB 2025-02-15 05:57:51,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34110.18 MB 2025-02-15 05:57:51,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-15 05:57:51,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:57:51,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32061.84 MB 2025-02-15 05:57:51,917 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 05:57:51,917 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:57:51,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:57:51,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:57:51,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:57:51,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:57:51,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21965.67 MB 2025-02-15 05:57:51,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30365.06 MB 2025-02-15 05:57:51,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-15 05:57:51,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34110.18 MB 2025-02-15 05:57:51,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44549.80 MB 2025-02-15 05:57:51,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-15 05:57:51,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30365.06 MB 2025-02-15 05:57:52,085 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 05:57:52,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:52,086 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:57:52,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:52,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:57:52,092 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:57:52,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:57:52,093 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:57:52,093 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:58:45,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:45,831 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:58:45,836 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:58:45,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:45,841 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 156, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:58:45,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:45,842 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 156, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:58:48,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:58:48,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:58:48,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-15 05:58:48,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:48,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14055.74 MB 2025-02-15 05:58:48,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14607.81 MB 2025-02-15 05:58:48,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.08 MB 2025-02-15 05:58:48,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52900.66 MB 2025-02-15 05:58:48,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18838.72 MB 2025-02-15 05:58:48,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34061.94 MB 2025-02-15 05:58:48,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23527.11 MB 2025-02-15 05:58:48,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:58:48,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:58:48,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:58:48,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:48,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14607.81 MB 2025-02-15 05:58:48,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14875.29 MB 2025-02-15 05:58:48,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.48 MB 2025-02-15 05:58:48,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18838.72 MB 2025-02-15 05:58:48,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18838.72 MB 2025-02-15 05:58:48,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:58:48,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16799.06 MB 2025-02-15 05:58:49,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:58:49,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:58:49,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.75 seconds 2025-02-15 05:58:49,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14875.29 MB 2025-02-15 05:58:49,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.32 MB 2025-02-15 05:58:49,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.03 MB 2025-02-15 05:58:49,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18838.72 MB 2025-02-15 05:58:49,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 05:58:49,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 05:58:49,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19045.98 MB 2025-02-15 05:58:49,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:58:49,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:58:49,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 05:58:49,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.26 MB 2025-02-15 05:58:49,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15819.00 MB 2025-02-15 05:58:49,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.74 MB 2025-02-15 05:58:49,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 05:58:49,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 05:58:49,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:58:49,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16371.80 MB 2025-02-15 05:58:49,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:58:49,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:58:49,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:58:49,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15819.00 MB 2025-02-15 05:58:49,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.36 MB 2025-02-15 05:58:49,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 874.36 MB 2025-02-15 05:58:49,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 05:58:49,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-15 05:58:49,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1845.49 MB 2025-02-15 05:58:49,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18859.78 MB 2025-02-15 05:58:49,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:58:49,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:58:49,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 05:58:49,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.26 MB 2025-02-15 05:58:49,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.36 MB 2025-02-15 05:58:49,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1611.10 MB 2025-02-15 05:58:49,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 05:58:49,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-15 05:58:49,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1845.49 MB 2025-02-15 05:58:49,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18859.78 MB 2025-02-15 05:58:49,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:58:49,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:58:49,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 05:58:49,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17291.44 MB 2025-02-15 05:58:49,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17590.57 MB 2025-02-15 05:58:49,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.13 MB 2025-02-15 05:58:49,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-15 05:58:49,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20371.73 MB 2025-02-15 05:58:49,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-15 05:58:49,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.77 MB 2025-02-15 05:58:49,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:58:49,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:58:49,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:58:49,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17751.61 MB 2025-02-15 05:58:49,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17963.47 MB 2025-02-15 05:58:49,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.87 MB 2025-02-15 05:58:49,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20371.73 MB 2025-02-15 05:58:49,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20371.73 MB 2025-02-15 05:58:49,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:58:49,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17977.60 MB 2025-02-15 05:58:49,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:58:49,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:58:49,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.36 seconds 2025-02-15 05:58:49,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13512.22 MB 2025-02-15 05:58:49,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18164.55 MB 2025-02-15 05:58:49,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4652.32 MB 2025-02-15 05:58:49,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52900.66 MB 2025-02-15 05:58:49,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20371.73 MB 2025-02-15 05:58:49,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32528.92 MB 2025-02-15 05:58:49,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18164.55 MB 2025-02-15 05:58:49,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:58:49,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:58:49,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:58:49,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18164.55 MB 2025-02-15 05:58:49,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17365.10 MB 2025-02-15 05:58:49,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -799.44 MB 2025-02-15 05:58:49,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20371.73 MB 2025-02-15 05:58:49,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20371.73 MB 2025-02-15 05:58:49,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:58:49,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19068.75 MB 2025-02-15 05:58:49,492 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:58:49,492 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:58:49,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:58:49,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:58:49,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:58:49,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:58:49,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17365.10 MB 2025-02-15 05:58:49,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25804.13 MB 2025-02-15 05:58:49,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 05:58:49,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20371.73 MB 2025-02-15 05:58:49,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30861.69 MB 2025-02-15 05:58:49,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 05:58:49,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25804.13 MB 2025-02-15 05:58:49,664 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:58:49,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:49,666 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:58:49,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:49,667 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:58:49,673 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:58:49,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:58:49,674 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:58:49,675 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:59:01,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:01,002 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 05:59:01,007 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 05:59:01,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:01,010 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1300, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 05:59:01,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:01,011 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1300, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 05:59:21,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 05:59:21,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 05:59:21,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.05 seconds 2025-02-15 05:59:21,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:21,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22027.31 MB 2025-02-15 05:59:21,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26628.47 MB 2025-02-15 05:59:21,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4601.15 MB 2025-02-15 05:59:21,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43446.70 MB 2025-02-15 05:59:21,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37859.89 MB 2025-02-15 05:59:21,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5586.81 MB 2025-02-15 05:59:21,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35575.55 MB 2025-02-15 05:59:21,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 05:59:21,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 05:59:21,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 05:59:21,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:21,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26628.47 MB 2025-02-15 05:59:21,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22536.14 MB 2025-02-15 05:59:21,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4092.32 MB 2025-02-15 05:59:21,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37859.89 MB 2025-02-15 05:59:21,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46867.15 MB 2025-02-15 05:59:21,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9007.27 MB 2025-02-15 05:59:21,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40169.60 MB 2025-02-15 05:59:23,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 05:59:23,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 05:59:23,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 05:59:23,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22536.14 MB 2025-02-15 05:59:23,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23066.99 MB 2025-02-15 05:59:23,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 05:59:23,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46867.15 MB 2025-02-15 05:59:23,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:59:23,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17802.72 MB 2025-02-15 05:59:23,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27045.53 MB 2025-02-15 05:59:23,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 05:59:23,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 05:59:23,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 05:59:23,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23066.99 MB 2025-02-15 05:59:23,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24956.52 MB 2025-02-15 05:59:23,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 05:59:23,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:59:23,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 05:59:23,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:59:23,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26373.95 MB 2025-02-15 05:59:23,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 05:59:23,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 05:59:23,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 05:59:23,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24956.52 MB 2025-02-15 05:59:23,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27198.38 MB 2025-02-15 05:59:23,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 05:59:23,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:59:23,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:59:23,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:59:23,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32742.66 MB 2025-02-15 05:59:23,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 05:59:23,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 05:59:23,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 05:59:23,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23066.99 MB 2025-02-15 05:59:23,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27198.38 MB 2025-02-15 05:59:23,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 05:59:23,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 05:59:23,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34726.74 MB 2025-02-15 05:59:23,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 05:59:23,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32742.66 MB 2025-02-15 05:59:23,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 05:59:23,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 05:59:23,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 05:59:23,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28731.92 MB 2025-02-15 05:59:23,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29498.92 MB 2025-02-15 05:59:23,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 05:59:23,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34726.74 MB 2025-02-15 05:59:23,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:59:23,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 05:59:23,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30206.71 MB 2025-02-15 05:59:23,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 05:59:23,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 05:59:23,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:59:23,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29911.81 MB 2025-02-15 05:59:23,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30140.44 MB 2025-02-15 05:59:23,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.63 MB 2025-02-15 05:59:23,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:59:23,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:59:23,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:59:23,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30374.66 MB 2025-02-15 05:59:23,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 05:59:23,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 05:59:23,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.51 seconds 2025-02-15 05:59:23,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.01 MB 2025-02-15 05:59:23,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.51 MB 2025-02-15 05:59:23,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12843.50 MB 2025-02-15 05:59:23,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43446.70 MB 2025-02-15 05:59:23,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:59:23,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8304.72 MB 2025-02-15 05:59:23,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30374.66 MB 2025-02-15 05:59:23,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 05:59:23,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 05:59:23,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 05:59:23,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30341.51 MB 2025-02-15 05:59:23,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22502.40 MB 2025-02-15 05:59:23,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7839.11 MB 2025-02-15 05:59:23,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:59:23,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 05:59:23,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 05:59:23,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32853.18 MB 2025-02-15 05:59:23,812 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 05:59:23,812 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 05:59:23,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 05:59:23,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 05:59:23,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 05:59:23,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 05:59:23,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22502.40 MB 2025-02-15 05:59:23,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30941.09 MB 2025-02-15 05:59:23,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-15 05:59:23,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 05:59:23,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39338.38 MB 2025-02-15 05:59:23,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 05:59:23,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30941.09 MB 2025-02-15 05:59:23,978 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 05:59:23,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:23,980 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 05:59:23,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:23,981 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 05:59:23,985 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 05:59:23,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 05:59:23,987 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 05:59:23,987 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:00:56,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:00:56,208 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:00:56,213 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:00:56,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:00:56,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 198, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:00:56,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:00:56,218 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 198, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:00:59,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:00:59,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:00:59,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-15 06:00:59,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:00:59,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14348.40 MB 2025-02-15 06:00:59,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15049.11 MB 2025-02-15 06:00:59,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 700.71 MB 2025-02-15 06:00:59,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47726.99 MB 2025-02-15 06:00:59,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-15 06:00:59,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27722.25 MB 2025-02-15 06:00:59,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24047.00 MB 2025-02-15 06:00:59,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:00:59,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:00:59,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:00:59,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:00:59,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15049.11 MB 2025-02-15 06:00:59,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15388.61 MB 2025-02-15 06:00:59,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.49 MB 2025-02-15 06:00:59,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-15 06:00:59,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-15 06:00:59,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:00:59,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17830.30 MB 2025-02-15 06:01:00,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:01:00,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:01:00,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-15 06:01:00,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15388.61 MB 2025-02-15 06:01:00,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15651.37 MB 2025-02-15 06:01:00,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-15 06:01:00,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-15 06:01:00,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-15 06:01:00,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 06:01:00,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19644.26 MB 2025-02-15 06:01:00,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:01:00,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:01:00,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:01:00,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15651.31 MB 2025-02-15 06:01:00,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16586.40 MB 2025-02-15 06:01:00,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 935.09 MB 2025-02-15 06:01:00,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 06:01:00,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-15 06:01:00,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:01:00,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17288.03 MB 2025-02-15 06:01:00,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:01:00,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:01:00,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:01:00,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16586.40 MB 2025-02-15 06:01:00,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17696.15 MB 2025-02-15 06:01:00,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.75 MB 2025-02-15 06:01:00,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 06:01:00,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21871.20 MB 2025-02-15 06:01:00,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2338.32 MB 2025-02-15 06:01:00,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20442.64 MB 2025-02-15 06:01:00,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:01:00,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:01:00,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 06:01:00,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15651.31 MB 2025-02-15 06:01:00,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17696.15 MB 2025-02-15 06:01:00,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2044.84 MB 2025-02-15 06:01:00,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-15 06:01:00,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21871.20 MB 2025-02-15 06:01:00,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2338.32 MB 2025-02-15 06:01:00,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20442.64 MB 2025-02-15 06:01:00,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:01:00,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:01:00,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:01:00,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18455.25 MB 2025-02-15 06:01:00,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.92 MB 2025-02-15 06:01:00,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.67 MB 2025-02-15 06:01:00,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21871.20 MB 2025-02-15 06:01:00,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22074.62 MB 2025-02-15 06:01:00,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 203.42 MB 2025-02-15 06:01:00,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19189.20 MB 2025-02-15 06:01:00,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:01:00,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:01:00,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:01:00,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19039.31 MB 2025-02-15 06:01:00,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19260.21 MB 2025-02-15 06:01:00,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.90 MB 2025-02-15 06:01:00,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22074.62 MB 2025-02-15 06:01:00,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22074.62 MB 2025-02-15 06:01:00,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:01:00,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.70 MB 2025-02-15 06:01:00,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:01:00,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:01:00,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.24 seconds 2025-02-15 06:01:00,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13658.55 MB 2025-02-15 06:01:00,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19461.20 MB 2025-02-15 06:01:00,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5802.65 MB 2025-02-15 06:01:00,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47726.99 MB 2025-02-15 06:01:00,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22074.62 MB 2025-02-15 06:01:00,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25652.36 MB 2025-02-15 06:01:00,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19461.20 MB 2025-02-15 06:01:00,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:01:00,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:01:00,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:01:00,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14695.58 MB 2025-02-15 06:01:00,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17708.50 MB 2025-02-15 06:01:00,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-15 06:01:00,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22074.62 MB 2025-02-15 06:01:00,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22074.62 MB 2025-02-15 06:01:00,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:01:00,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18009.76 MB 2025-02-15 06:01:00,751 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 06:01:00,751 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 06:01:00,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:01:00,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:01:00,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:01:00,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:01:00,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17708.50 MB 2025-02-15 06:01:00,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26144.10 MB 2025-02-15 06:01:00,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 06:01:00,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22074.62 MB 2025-02-15 06:01:00,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32560.38 MB 2025-02-15 06:01:00,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 06:01:00,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26144.10 MB 2025-02-15 06:01:00,921 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 06:01:00,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:01:00,922 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:01:00,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:01:00,923 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:01:00,928 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:01:00,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:01:00,929 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:01:00,929 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 06:02:09,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:09,059 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:02:09,064 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:02:09,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:09,068 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2686, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:02:09,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:09,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2686, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:02:50,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:02:50,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:02:50,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.35 seconds 2025-02-15 06:02:50,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:50,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31685.18 MB 2025-02-15 06:02:50,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41191.57 MB 2025-02-15 06:02:50,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9506.39 MB 2025-02-15 06:02:50,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59672.36 MB 2025-02-15 06:02:50,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44707.09 MB 2025-02-15 06:02:50,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14965.28 MB 2025-02-15 06:02:50,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50697.18 MB 2025-02-15 06:02:50,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:02:50,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:02:50,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:02:50,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:50,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41191.57 MB 2025-02-15 06:02:50,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29742.58 MB 2025-02-15 06:02:50,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11449.00 MB 2025-02-15 06:02:50,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44707.09 MB 2025-02-15 06:02:50,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53886.32 MB 2025-02-15 06:02:50,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9179.23 MB 2025-02-15 06:02:50,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52468.82 MB 2025-02-15 06:02:52,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:02:52,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:02:52,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:02:52,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29742.58 MB 2025-02-15 06:02:52,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30273.42 MB 2025-02-15 06:02:52,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:02:52,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53886.32 MB 2025-02-15 06:02:52,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32570.87 MB 2025-02-15 06:02:52,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21315.45 MB 2025-02-15 06:02:52,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34253.00 MB 2025-02-15 06:02:52,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:02:52,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:02:52,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:02:52,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30273.42 MB 2025-02-15 06:02:52,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32162.86 MB 2025-02-15 06:02:52,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.44 MB 2025-02-15 06:02:52,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32570.87 MB 2025-02-15 06:02:52,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35402.02 MB 2025-02-15 06:02:52,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:02:52,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33580.29 MB 2025-02-15 06:02:52,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:02:52,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:02:52,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:02:52,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32162.86 MB 2025-02-15 06:02:52,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34404.72 MB 2025-02-15 06:02:52,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:02:52,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35402.02 MB 2025-02-15 06:02:52,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41536.19 MB 2025-02-15 06:02:52,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 06:02:52,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39949.00 MB 2025-02-15 06:02:52,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:02:52,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:02:52,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 06:02:52,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30273.42 MB 2025-02-15 06:02:52,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34404.72 MB 2025-02-15 06:02:52,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.30 MB 2025-02-15 06:02:52,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32570.87 MB 2025-02-15 06:02:52,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41536.19 MB 2025-02-15 06:02:52,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 06:02:52,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39949.00 MB 2025-02-15 06:02:52,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:02:52,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:02:52,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:02:52,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35938.26 MB 2025-02-15 06:02:52,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36705.26 MB 2025-02-15 06:02:52,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:02:52,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41536.19 MB 2025-02-15 06:02:52,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 06:02:52,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:02:52,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37413.05 MB 2025-02-15 06:02:52,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:02:52,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:02:52,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:02:52,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37118.15 MB 2025-02-15 06:02:52,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37344.60 MB 2025-02-15 06:02:52,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.45 MB 2025-02-15 06:02:52,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 06:02:52,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 06:02:52,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:02:52,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37567.69 MB 2025-02-15 06:02:52,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:02:52,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:02:52,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.89 seconds 2025-02-15 06:02:52,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:52,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22326.95 MB 2025-02-15 06:02:52,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37545.67 MB 2025-02-15 06:02:52,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15218.72 MB 2025-02-15 06:02:52,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50310.68 MB 2025-02-15 06:02:52,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 06:02:52,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8357.15 MB 2025-02-15 06:02:52,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37567.69 MB 2025-02-15 06:02:53,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:02:53,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:02:53,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:02:53,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:53,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37545.67 MB 2025-02-15 06:02:53,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27331.24 MB 2025-02-15 06:02:53,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10214.43 MB 2025-02-15 06:02:53,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 06:02:53,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 06:02:53,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:02:53,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40057.34 MB 2025-02-15 06:02:53,251 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:02:53,251 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:02:53,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:02:53,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:02:53,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:02:53,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:02:53,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27331.24 MB 2025-02-15 06:02:53,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35769.93 MB 2025-02-15 06:02:53,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-15 06:02:53,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 06:02:53,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46149.93 MB 2025-02-15 06:02:53,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 06:02:53,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35769.93 MB 2025-02-15 06:02:53,419 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:02:53,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:53,421 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:02:53,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:53,422 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:02:53,426 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:02:53,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:02:53,428 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:02:53,428 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:03:02,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:02,284 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:03:02,288 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:03:02,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:02,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:03:02,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:02,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:03:29,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:03:29,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:03:29,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.23 seconds 2025-02-15 06:03:29,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:29,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.24 MB 2025-02-15 06:03:29,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31272.87 MB 2025-02-15 06:03:29,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6165.63 MB 2025-02-15 06:03:29,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58734.94 MB 2025-02-15 06:03:29,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39835.40 MB 2025-02-15 06:03:29,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18899.53 MB 2025-02-15 06:03:29,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40240.92 MB 2025-02-15 06:03:29,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:03:29,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:03:29,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:03:29,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:29,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31272.87 MB 2025-02-15 06:03:29,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24833.97 MB 2025-02-15 06:03:29,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6438.90 MB 2025-02-15 06:03:29,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39835.40 MB 2025-02-15 06:03:29,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50272.93 MB 2025-02-15 06:03:29,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10437.53 MB 2025-02-15 06:03:29,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46969.78 MB 2025-02-15 06:03:31,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:03:31,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:03:31,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:03:31,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24833.97 MB 2025-02-15 06:03:31,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25364.81 MB 2025-02-15 06:03:31,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:03:31,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50272.93 MB 2025-02-15 06:03:31,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-15 06:03:31,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20797.46 MB 2025-02-15 06:03:31,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29344.39 MB 2025-02-15 06:03:31,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:03:31,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:03:31,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:03:31,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-15 06:03:31,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27254.34 MB 2025-02-15 06:03:31,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:03:31,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-15 06:03:31,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-15 06:03:31,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:03:31,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28671.77 MB 2025-02-15 06:03:31,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:03:31,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:03:31,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:03:31,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27254.34 MB 2025-02-15 06:03:31,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-15 06:03:31,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:03:31,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-15 06:03:31,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37025.22 MB 2025-02-15 06:03:31,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:03:31,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-15 06:03:31,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:03:31,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:03:31,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:03:31,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-15 06:03:31,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-15 06:03:31,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:03:31,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-15 06:03:31,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37025.22 MB 2025-02-15 06:03:31,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 06:03:31,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-15 06:03:31,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:03:31,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:03:31,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:03:31,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31029.74 MB 2025-02-15 06:03:31,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31796.74 MB 2025-02-15 06:03:31,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:03:31,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37025.22 MB 2025-02-15 06:03:31,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37442.55 MB 2025-02-15 06:03:31,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:03:31,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32504.53 MB 2025-02-15 06:03:31,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:03:31,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:03:31,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:03:31,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.63 MB 2025-02-15 06:03:31,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32437.78 MB 2025-02-15 06:03:31,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-15 06:03:31,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37442.55 MB 2025-02-15 06:03:31,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37442.55 MB 2025-02-15 06:03:31,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:31,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.11 MB 2025-02-15 06:03:31,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:03:31,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:03:31,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.70 seconds 2025-02-15 06:03:31,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:31,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.97 MB 2025-02-15 06:03:31,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32638.78 MB 2025-02-15 06:03:31,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13600.80 MB 2025-02-15 06:03:31,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58734.94 MB 2025-02-15 06:03:31,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37442.55 MB 2025-02-15 06:03:31,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21292.38 MB 2025-02-15 06:03:31,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.11 MB 2025-02-15 06:03:32,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:03:32,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:03:32,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:03:32,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:32,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32638.78 MB 2025-02-15 06:03:32,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24041.22 MB 2025-02-15 06:03:32,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8597.56 MB 2025-02-15 06:03:32,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37442.55 MB 2025-02-15 06:03:32,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37442.55 MB 2025-02-15 06:03:32,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:32,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35149.52 MB 2025-02-15 06:03:32,292 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 06:03:32,292 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:03:32,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:03:32,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:03:32,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:03:32,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:32,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24041.22 MB 2025-02-15 06:03:32,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32476.81 MB 2025-02-15 06:03:32,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 06:03:32,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37442.55 MB 2025-02-15 06:03:32,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45831.16 MB 2025-02-15 06:03:32,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 06:03:32,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32476.81 MB 2025-02-15 06:03:32,456 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 06:03:32,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:32,458 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:03:32,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:32,459 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:03:32,463 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:03:32,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:32,464 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:03:32,465 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:03:41,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:41,484 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:03:41,489 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:03:41,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:41,493 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 171, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:03:41,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:41,494 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 171, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:03:44,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:03:44,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:03:44,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.69 seconds 2025-02-15 06:03:44,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:44,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14160.26 MB 2025-02-15 06:03:44,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14765.42 MB 2025-02-15 06:03:44,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 605.16 MB 2025-02-15 06:03:44,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54219.77 MB 2025-02-15 06:03:44,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 06:03:44,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36798.73 MB 2025-02-15 06:03:44,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23631.63 MB 2025-02-15 06:03:44,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:03:44,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:03:44,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:03:44,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:44,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14765.42 MB 2025-02-15 06:03:44,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15016.69 MB 2025-02-15 06:03:44,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 251.27 MB 2025-02-15 06:03:44,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 06:03:44,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18582.86 MB 2025-02-15 06:03:44,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1161.82 MB 2025-02-15 06:03:44,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17083.29 MB 2025-02-15 06:03:45,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:03:45,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:03:45,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-15 06:03:45,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15016.69 MB 2025-02-15 06:03:45,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15235.66 MB 2025-02-15 06:03:45,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-15 06:03:45,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18582.86 MB 2025-02-15 06:03:45,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18582.86 MB 2025-02-15 06:03:45,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:45,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19187.38 MB 2025-02-15 06:03:45,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:03:45,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:03:45,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:03:45,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15235.59 MB 2025-02-15 06:03:45,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16014.84 MB 2025-02-15 06:03:45,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 779.24 MB 2025-02-15 06:03:45,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18582.86 MB 2025-02-15 06:03:45,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18582.86 MB 2025-02-15 06:03:45,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:45,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16599.53 MB 2025-02-15 06:03:45,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:03:45,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:03:45,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:03:45,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16014.84 MB 2025-02-15 06:03:45,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16939.64 MB 2025-02-15 06:03:45,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 924.80 MB 2025-02-15 06:03:45,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18582.86 MB 2025-02-15 06:03:45,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20533.22 MB 2025-02-15 06:03:45,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1950.35 MB 2025-02-15 06:03:45,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19230.55 MB 2025-02-15 06:03:45,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:03:45,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:03:45,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:03:45,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15235.59 MB 2025-02-15 06:03:45,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16939.64 MB 2025-02-15 06:03:45,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1704.05 MB 2025-02-15 06:03:45,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18582.86 MB 2025-02-15 06:03:45,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20533.22 MB 2025-02-15 06:03:45,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1950.35 MB 2025-02-15 06:03:45,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19230.55 MB 2025-02-15 06:03:45,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:03:45,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:03:45,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:03:45,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17572.23 MB 2025-02-15 06:03:45,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17889.40 MB 2025-02-15 06:03:45,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.17 MB 2025-02-15 06:03:45,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20533.22 MB 2025-02-15 06:03:45,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-15 06:03:45,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 06:03:45,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18188.55 MB 2025-02-15 06:03:45,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:03:45,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:03:45,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:03:45,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18059.73 MB 2025-02-15 06:03:45,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18276.25 MB 2025-02-15 06:03:45,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.52 MB 2025-02-15 06:03:45,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20703.08 MB 2025-02-15 06:03:45,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-15 06:03:45,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:45,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18290.01 MB 2025-02-15 06:03:45,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:03:45,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:03:45,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.69 seconds 2025-02-15 06:03:45,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13564.48 MB 2025-02-15 06:03:45,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18477.10 MB 2025-02-15 06:03:45,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4912.62 MB 2025-02-15 06:03:45,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54219.77 MB 2025-02-15 06:03:45,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-15 06:03:45,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33516.68 MB 2025-02-15 06:03:45,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18477.10 MB 2025-02-15 06:03:45,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:03:45,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:03:45,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:03:45,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18477.10 MB 2025-02-15 06:03:45,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17454.96 MB 2025-02-15 06:03:45,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1022.14 MB 2025-02-15 06:03:45,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20703.08 MB 2025-02-15 06:03:45,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-15 06:03:45,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:03:45,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19179.16 MB 2025-02-15 06:03:45,484 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 06:03:45,485 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:03:45,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:03:45,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:03:45,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 06:03:45,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:03:45,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17454.96 MB 2025-02-15 06:03:45,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25879.91 MB 2025-02-15 06:03:45,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 06:03:45,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20703.08 MB 2025-02-15 06:03:45,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31174.16 MB 2025-02-15 06:03:45,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 06:03:45,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25879.91 MB 2025-02-15 06:03:45,651 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 06:03:45,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:45,652 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:03:45,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:45,653 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:03:45,658 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:03:45,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:03:45,659 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:03:45,659 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:04:40,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:40,029 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:04:40,034 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:04:40,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:40,038 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:04:40,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:40,039 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:04:43,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:04:43,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:04:43,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-15 06:04:43,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:43,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14438.99 MB 2025-02-15 06:04:43,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15185.71 MB 2025-02-15 06:04:43,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-15 06:04:43,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39550.19 MB 2025-02-15 06:04:43,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19677.58 MB 2025-02-15 06:04:43,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19872.61 MB 2025-02-15 06:04:43,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24136.85 MB 2025-02-15 06:04:43,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:04:43,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:04:43,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:04:43,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:43,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15185.71 MB 2025-02-15 06:04:43,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15266.57 MB 2025-02-15 06:04:43,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.86 MB 2025-02-15 06:04:43,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19677.58 MB 2025-02-15 06:04:43,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19677.58 MB 2025-02-15 06:04:43,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:04:43,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17587.64 MB 2025-02-15 06:04:44,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:04:44,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:04:44,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 06:04:44,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15266.57 MB 2025-02-15 06:04:44,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15493.50 MB 2025-02-15 06:04:44,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.93 MB 2025-02-15 06:04:44,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19677.58 MB 2025-02-15 06:04:44,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19287.51 MB 2025-02-15 06:04:44,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -390.07 MB 2025-02-15 06:04:44,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19437.26 MB 2025-02-15 06:04:44,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:04:44,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:04:44,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:04:44,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15493.44 MB 2025-02-15 06:04:44,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16301.02 MB 2025-02-15 06:04:44,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.58 MB 2025-02-15 06:04:44,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 06:04:44,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19287.51 MB 2025-02-15 06:04:44,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:04:44,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16906.97 MB 2025-02-15 06:04:44,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:04:44,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:04:44,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:04:44,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16301.02 MB 2025-02-15 06:04:44,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.45 MB 2025-02-15 06:04:44,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 958.43 MB 2025-02-15 06:04:44,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 06:04:44,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21109.93 MB 2025-02-15 06:04:44,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1822.43 MB 2025-02-15 06:04:44,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19629.59 MB 2025-02-15 06:04:44,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:04:44,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:04:44,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:04:44,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15493.44 MB 2025-02-15 06:04:44,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.45 MB 2025-02-15 06:04:44,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1766.01 MB 2025-02-15 06:04:44,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 06:04:44,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21109.93 MB 2025-02-15 06:04:44,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1822.43 MB 2025-02-15 06:04:44,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19629.59 MB 2025-02-15 06:04:44,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:04:44,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:04:44,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:04:44,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17915.04 MB 2025-02-15 06:04:44,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18242.93 MB 2025-02-15 06:04:44,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 327.89 MB 2025-02-15 06:04:44,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21109.93 MB 2025-02-15 06:04:44,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21284.00 MB 2025-02-15 06:04:44,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 06:04:44,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18553.64 MB 2025-02-15 06:04:44,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:04:44,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:04:44,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:04:44,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18419.45 MB 2025-02-15 06:04:44,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18628.21 MB 2025-02-15 06:04:44,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.76 MB 2025-02-15 06:04:44,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21284.00 MB 2025-02-15 06:04:44,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21284.00 MB 2025-02-15 06:04:44,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:04:44,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18650.94 MB 2025-02-15 06:04:44,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:04:44,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:04:44,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.27 seconds 2025-02-15 06:04:44,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13703.85 MB 2025-02-15 06:04:44,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18829.08 MB 2025-02-15 06:04:44,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5125.24 MB 2025-02-15 06:04:44,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39550.19 MB 2025-02-15 06:04:44,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21284.00 MB 2025-02-15 06:04:44,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18266.19 MB 2025-02-15 06:04:44,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18829.08 MB 2025-02-15 06:04:44,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:04:44,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:04:44,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:04:44,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18829.08 MB 2025-02-15 06:04:44,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17624.08 MB 2025-02-15 06:04:44,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1205.00 MB 2025-02-15 06:04:44,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21284.00 MB 2025-02-15 06:04:44,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21284.00 MB 2025-02-15 06:04:44,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:04:44,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19063.95 MB 2025-02-15 06:04:44,595 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 06:04:44,596 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:04:44,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:04:44,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:04:44,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:04:44,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:04:44,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17624.08 MB 2025-02-15 06:04:44,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26054.75 MB 2025-02-15 06:04:44,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 06:04:44,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21284.00 MB 2025-02-15 06:04:44,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29666.31 MB 2025-02-15 06:04:44,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 06:04:44,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26054.75 MB 2025-02-15 06:04:44,759 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 06:04:44,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:44,761 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:04:44,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:44,762 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:04:44,766 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:04:44,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:44,767 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:04:44,768 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:04:54,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:54,100 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:04:54,104 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:04:54,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:54,108 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1248, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:04:54,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:04:54,109 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1248, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:05:13,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:05:13,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:05:13,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.33 seconds 2025-02-15 06:05:13,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:13,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21664.97 MB 2025-02-15 06:05:13,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26081.57 MB 2025-02-15 06:05:13,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4416.60 MB 2025-02-15 06:05:13,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42238.74 MB 2025-02-15 06:05:13,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37656.46 MB 2025-02-15 06:05:13,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4582.28 MB 2025-02-15 06:05:13,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34986.71 MB 2025-02-15 06:05:13,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:05:13,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:05:13,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:05:13,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:13,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26081.57 MB 2025-02-15 06:05:13,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22265.81 MB 2025-02-15 06:05:13,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3815.76 MB 2025-02-15 06:05:13,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37656.46 MB 2025-02-15 06:05:13,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44224.74 MB 2025-02-15 06:05:13,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6568.28 MB 2025-02-15 06:05:13,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39148.08 MB 2025-02-15 06:05:15,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:05:15,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:05:15,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:05:15,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22265.81 MB 2025-02-15 06:05:15,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22796.65 MB 2025-02-15 06:05:15,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:05:15,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44224.74 MB 2025-02-15 06:05:15,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29049.75 MB 2025-02-15 06:05:15,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15174.99 MB 2025-02-15 06:05:15,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26775.20 MB 2025-02-15 06:05:15,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:05:15,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:05:15,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:05:15,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-15 06:05:15,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24686.19 MB 2025-02-15 06:05:15,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:05:15,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29049.75 MB 2025-02-15 06:05:15,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29049.75 MB 2025-02-15 06:05:15,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:15,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26103.62 MB 2025-02-15 06:05:15,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:05:15,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:05:15,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:05:15,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24686.19 MB 2025-02-15 06:05:15,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-15 06:05:15,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:05:15,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29049.75 MB 2025-02-15 06:05:15,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 06:05:15,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:05:15,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-15 06:05:15,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:05:15,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:05:15,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:05:15,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-15 06:05:15,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-15 06:05:15,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:05:15,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29049.75 MB 2025-02-15 06:05:15,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 06:05:15,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:05:15,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-15 06:05:15,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:05:15,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:05:15,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:05:15,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28461.59 MB 2025-02-15 06:05:15,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29228.59 MB 2025-02-15 06:05:15,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:05:15,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-15 06:05:15,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 06:05:15,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:05:15,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29936.38 MB 2025-02-15 06:05:15,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:05:15,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:05:15,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:05:15,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29641.48 MB 2025-02-15 06:05:15,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.12 MB 2025-02-15 06:05:15,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-15 06:05:15,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 06:05:15,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 06:05:15,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:15,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30107.16 MB 2025-02-15 06:05:15,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:05:15,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:05:15,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.74 seconds 2025-02-15 06:05:15,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:15,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17316.84 MB 2025-02-15 06:05:15,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30070.97 MB 2025-02-15 06:05:15,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12754.13 MB 2025-02-15 06:05:15,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42238.74 MB 2025-02-15 06:05:15,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 06:05:15,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7109.35 MB 2025-02-15 06:05:15,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30107.16 MB 2025-02-15 06:05:16,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:05:16,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:05:16,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:05:16,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:16,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30070.97 MB 2025-02-15 06:05:16,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22313.52 MB 2025-02-15 06:05:16,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7757.45 MB 2025-02-15 06:05:16,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 06:05:16,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-15 06:05:16,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:16,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32576.19 MB 2025-02-15 06:05:16,139 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 06:05:16,140 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:05:16,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:05:16,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:05:16,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:05:16,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:16,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22313.52 MB 2025-02-15 06:05:16,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30730.64 MB 2025-02-15 06:05:16,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-15 06:05:16,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-15 06:05:16,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39313.21 MB 2025-02-15 06:05:16,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 06:05:16,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30730.64 MB 2025-02-15 06:05:16,303 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 06:05:16,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:16,304 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:05:16,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:16,305 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:05:16,310 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:05:16,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:16,311 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:05:16,311 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:05:25,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:25,761 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:05:25,766 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:05:25,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:25,770 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 164, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:05:25,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:25,771 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 164, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:05:28,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:05:28,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:05:28,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-15 06:05:28,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:28,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14111.48 MB 2025-02-15 06:05:28,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14691.87 MB 2025-02-15 06:05:28,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 580.39 MB 2025-02-15 06:05:28,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47680.85 MB 2025-02-15 06:05:28,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 06:05:28,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30259.81 MB 2025-02-15 06:05:28,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23582.86 MB 2025-02-15 06:05:28,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:05:28,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:05:28,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:05:28,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:28,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14691.87 MB 2025-02-15 06:05:28,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14953.75 MB 2025-02-15 06:05:28,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 261.88 MB 2025-02-15 06:05:28,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 06:05:28,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 06:05:28,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1132.46 MB 2025-02-15 06:05:28,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16955.10 MB 2025-02-15 06:05:29,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:05:29,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:05:29,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-15 06:05:29,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14953.75 MB 2025-02-15 06:05:29,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15167.42 MB 2025-02-15 06:05:29,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 06:05:29,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 06:05:29,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 06:05:29,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:29,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19124.44 MB 2025-02-15 06:05:29,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:05:29,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:05:29,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:05:29,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15167.35 MB 2025-02-15 06:05:29,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15927.70 MB 2025-02-15 06:05:29,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 06:05:29,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 06:05:29,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 06:05:29,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:29,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16498.96 MB 2025-02-15 06:05:29,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:05:29,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:05:29,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:05:29,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15927.70 MB 2025-02-15 06:05:29,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16830.09 MB 2025-02-15 06:05:29,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 06:05:29,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 06:05:29,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20080.23 MB 2025-02-15 06:05:29,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-15 06:05:29,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19062.37 MB 2025-02-15 06:05:29,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:05:29,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:05:29,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 06:05:29,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15167.35 MB 2025-02-15 06:05:29,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16830.09 MB 2025-02-15 06:05:29,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 06:05:29,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 06:05:29,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20080.23 MB 2025-02-15 06:05:29,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-15 06:05:29,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19062.37 MB 2025-02-15 06:05:29,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:05:29,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:05:29,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:05:29,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17447.34 MB 2025-02-15 06:05:29,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17756.80 MB 2025-02-15 06:05:29,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.46 MB 2025-02-15 06:05:29,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20080.23 MB 2025-02-15 06:05:29,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20248.00 MB 2025-02-15 06:05:29,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 06:05:29,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18049.74 MB 2025-02-15 06:05:29,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:05:29,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:05:29,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:05:29,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17923.00 MB 2025-02-15 06:05:29,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18150.57 MB 2025-02-15 06:05:29,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.58 MB 2025-02-15 06:05:29,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20248.00 MB 2025-02-15 06:05:29,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20248.00 MB 2025-02-15 06:05:29,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:29,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18177.03 MB 2025-02-15 06:05:29,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:05:29,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:05:29,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.66 seconds 2025-02-15 06:05:29,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13540.10 MB 2025-02-15 06:05:29,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18351.65 MB 2025-02-15 06:05:29,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4811.55 MB 2025-02-15 06:05:29,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47680.85 MB 2025-02-15 06:05:29,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20248.00 MB 2025-02-15 06:05:29,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27432.85 MB 2025-02-15 06:05:29,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18351.65 MB 2025-02-15 06:05:29,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:05:29,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:05:29,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 06:05:29,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18351.65 MB 2025-02-15 06:05:29,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17417.31 MB 2025-02-15 06:05:29,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -934.33 MB 2025-02-15 06:05:29,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20248.00 MB 2025-02-15 06:05:29,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20248.00 MB 2025-02-15 06:05:29,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:05:29,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19155.38 MB 2025-02-15 06:05:29,743 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:05:29,744 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:05:29,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:05:29,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:05:29,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:05:29,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:05:29,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17417.31 MB 2025-02-15 06:05:29,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25856.34 MB 2025-02-15 06:05:29,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:05:29,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20248.00 MB 2025-02-15 06:05:29,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30737.96 MB 2025-02-15 06:05:29,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 06:05:29,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25856.34 MB 2025-02-15 06:05:30,004 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:05:30,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:30,006 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:05:30,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:30,008 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:05:30,015 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:05:30,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:05:30,018 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:05:30,018 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:06:18,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:18,105 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:06:18,110 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:06:18,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:18,113 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:06:18,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:18,114 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:06:20,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:06:20,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:06:20,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-15 06:06:20,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:20,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-15 06:06:20,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-15 06:06:20,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-15 06:06:20,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43322.97 MB 2025-02-15 06:06:20,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18004.05 MB 2025-02-15 06:06:20,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25318.92 MB 2025-02-15 06:06:20,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.28 MB 2025-02-15 06:06:20,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:06:20,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:06:20,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:06:20,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:20,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-15 06:06:20,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15067.17 MB 2025-02-15 06:06:20,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.17 MB 2025-02-15 06:06:20,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18004.05 MB 2025-02-15 06:06:20,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18868.08 MB 2025-02-15 06:06:20,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 864.03 MB 2025-02-15 06:06:20,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17185.14 MB 2025-02-15 06:06:21,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:06:21,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:06:21,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 06:06:21,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15067.17 MB 2025-02-15 06:06:21,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15284.81 MB 2025-02-15 06:06:21,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-15 06:06:21,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18868.08 MB 2025-02-15 06:06:21,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18889.05 MB 2025-02-15 06:06:21,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20.97 MB 2025-02-15 06:06:21,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19237.86 MB 2025-02-15 06:06:21,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:06:21,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:06:21,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:06:21,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15284.75 MB 2025-02-15 06:06:21,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16059.27 MB 2025-02-15 06:06:21,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-15 06:06:21,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 06:06:21,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18889.05 MB 2025-02-15 06:06:21,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:06:21,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16640.42 MB 2025-02-15 06:06:21,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:06:21,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:06:21,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:06:21,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16059.27 MB 2025-02-15 06:06:21,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16978.47 MB 2025-02-15 06:06:21,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.20 MB 2025-02-15 06:06:21,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 06:06:21,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20635.98 MB 2025-02-15 06:06:21,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1746.93 MB 2025-02-15 06:06:21,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19255.78 MB 2025-02-15 06:06:21,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:06:21,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:06:21,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 06:06:21,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15284.75 MB 2025-02-15 06:06:21,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16978.47 MB 2025-02-15 06:06:21,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.72 MB 2025-02-15 06:06:21,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 06:06:21,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20635.98 MB 2025-02-15 06:06:21,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1746.93 MB 2025-02-15 06:06:21,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19255.78 MB 2025-02-15 06:06:21,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:06:21,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:06:21,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:06:21,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17607.22 MB 2025-02-15 06:06:21,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17921.69 MB 2025-02-15 06:06:21,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-15 06:06:21,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20635.98 MB 2025-02-15 06:06:21,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20805.84 MB 2025-02-15 06:06:21,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 06:06:21,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18220.93 MB 2025-02-15 06:06:21,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:06:21,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:06:21,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:06:21,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18090.98 MB 2025-02-15 06:06:21,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18303.26 MB 2025-02-15 06:06:21,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.27 MB 2025-02-15 06:06:21,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20805.84 MB 2025-02-15 06:06:21,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20805.84 MB 2025-02-15 06:06:21,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:06:21,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18330.09 MB 2025-02-15 06:06:21,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:06:21,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:06:21,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.81 seconds 2025-02-15 06:06:21,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:21,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-15 06:06:21,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18504.28 MB 2025-02-15 06:06:21,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.47 MB 2025-02-15 06:06:21,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43322.97 MB 2025-02-15 06:06:21,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20805.84 MB 2025-02-15 06:06:21,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22517.12 MB 2025-02-15 06:06:21,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18504.28 MB 2025-02-15 06:06:22,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:06:22,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:06:22,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 06:06:22,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:22,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18504.28 MB 2025-02-15 06:06:22,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17492.16 MB 2025-02-15 06:06:22,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1012.12 MB 2025-02-15 06:06:22,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20805.84 MB 2025-02-15 06:06:22,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20805.84 MB 2025-02-15 06:06:22,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:06:22,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19207.37 MB 2025-02-15 06:06:22,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 06:06:22,214 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:06:22,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:06:22,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:06:22,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:06:22,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:06:22,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17492.16 MB 2025-02-15 06:06:22,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25929.63 MB 2025-02-15 06:06:22,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 06:06:22,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20805.84 MB 2025-02-15 06:06:22,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29194.45 MB 2025-02-15 06:06:22,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 06:06:22,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25929.63 MB 2025-02-15 06:06:22,380 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 06:06:22,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:22,382 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:06:22,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:22,383 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:06:22,387 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:06:22,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:06:22,388 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:06:22,389 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:08:00,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:00,370 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:08:00,375 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:08:00,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:00,379 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1129, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:08:00,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:00,380 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1129, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:08:17,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:08:17,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:08:17,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.24 seconds 2025-02-15 06:08:17,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:17,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20835.76 MB 2025-02-15 06:08:17,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24831.23 MB 2025-02-15 06:08:17,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3995.47 MB 2025-02-15 06:08:17,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37583.06 MB 2025-02-15 06:08:17,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28863.10 MB 2025-02-15 06:08:17,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8719.96 MB 2025-02-15 06:08:17,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33704.52 MB 2025-02-15 06:08:17,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:08:17,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:08:17,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:08:17,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:17,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24831.23 MB 2025-02-15 06:08:17,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21648.22 MB 2025-02-15 06:08:17,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3183.01 MB 2025-02-15 06:08:17,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28863.10 MB 2025-02-15 06:08:17,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38694.55 MB 2025-02-15 06:08:17,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9831.45 MB 2025-02-15 06:08:17,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36762.39 MB 2025-02-15 06:08:19,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:08:19,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:08:19,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:08:19,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:19,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21648.22 MB 2025-02-15 06:08:19,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22179.06 MB 2025-02-15 06:08:19,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:08:19,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38694.55 MB 2025-02-15 06:08:19,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26990.35 MB 2025-02-15 06:08:19,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11704.21 MB 2025-02-15 06:08:19,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26157.61 MB 2025-02-15 06:08:19,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:08:19,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:08:19,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:08:19,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:19,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22179.06 MB 2025-02-15 06:08:19,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24068.59 MB 2025-02-15 06:08:19,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:08:19,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 06:08:19,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27934.06 MB 2025-02-15 06:08:19,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:08:19,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25486.02 MB 2025-02-15 06:08:19,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:08:19,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:08:19,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:08:19,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:19,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24068.59 MB 2025-02-15 06:08:19,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26310.45 MB 2025-02-15 06:08:19,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:08:19,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27934.06 MB 2025-02-15 06:08:19,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 06:08:19,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:08:19,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31854.73 MB 2025-02-15 06:08:19,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:08:19,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:08:19,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:08:19,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:19,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22179.06 MB 2025-02-15 06:08:19,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26310.45 MB 2025-02-15 06:08:19,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:08:19,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 06:08:19,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 06:08:19,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:08:19,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31854.73 MB 2025-02-15 06:08:20,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:08:20,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:08:20,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:08:20,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:20,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27843.99 MB 2025-02-15 06:08:20,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28610.99 MB 2025-02-15 06:08:20,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:08:20,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-15 06:08:20,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 06:08:20,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:08:20,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29318.78 MB 2025-02-15 06:08:20,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:08:20,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:08:20,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:08:20,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:20,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29023.88 MB 2025-02-15 06:08:20,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29251.91 MB 2025-02-15 06:08:20,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-15 06:08:20,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 06:08:20,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 06:08:20,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:08:20,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.31 MB 2025-02-15 06:08:20,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:08:20,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:08:20,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.66 seconds 2025-02-15 06:08:20,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:20,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16902.23 MB 2025-02-15 06:08:20,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29452.76 MB 2025-02-15 06:08:20,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12550.53 MB 2025-02-15 06:08:20,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37583.06 MB 2025-02-15 06:08:20,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 06:08:20,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3569.35 MB 2025-02-15 06:08:20,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.31 MB 2025-02-15 06:08:20,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:08:20,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:08:20,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 06:08:20,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:20,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29452.76 MB 2025-02-15 06:08:20,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21889.29 MB 2025-02-15 06:08:20,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7563.46 MB 2025-02-15 06:08:20,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 06:08:20,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 06:08:20,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:08:20,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31949.68 MB 2025-02-15 06:08:20,326 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 06:08:20,326 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:08:20,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:08:20,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:08:20,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:08:20,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:08:20,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21889.29 MB 2025-02-15 06:08:20,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30278.44 MB 2025-02-15 06:08:20,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 06:08:20,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 06:08:20,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42356.18 MB 2025-02-15 06:08:20,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 06:08:20,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30278.44 MB 2025-02-15 06:08:20,490 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 06:08:20,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:20,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:08:20,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:20,492 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:08:20,497 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:08:20,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:08:20,498 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:08:20,498 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:09:31,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:09:31,444 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:09:31,450 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:09:31,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:09:31,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:09:31,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:09:31,455 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:10:02,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:10:02,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:10:02,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.78 seconds 2025-02-15 06:10:02,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:02,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26918.96 MB 2025-02-15 06:10:02,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34003.93 MB 2025-02-15 06:10:02,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7084.97 MB 2025-02-15 06:10:02,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-15 06:10:02,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40252.74 MB 2025-02-15 06:10:02,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10445.91 MB 2025-02-15 06:10:02,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42958.61 MB 2025-02-15 06:10:02,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:10:02,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:10:02,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:10:02,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:02,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34003.93 MB 2025-02-15 06:10:02,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26185.62 MB 2025-02-15 06:10:02,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7818.30 MB 2025-02-15 06:10:02,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40252.74 MB 2025-02-15 06:10:02,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55069.11 MB 2025-02-15 06:10:02,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14816.38 MB 2025-02-15 06:10:02,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54159.79 MB 2025-02-15 06:10:04,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:10:04,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:10:04,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:10:04,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26185.62 MB 2025-02-15 06:10:04,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26716.47 MB 2025-02-15 06:10:04,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:10:04,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55069.11 MB 2025-02-15 06:10:04,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30410.80 MB 2025-02-15 06:10:04,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24658.31 MB 2025-02-15 06:10:04,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30696.05 MB 2025-02-15 06:10:04,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:10:04,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:10:04,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:10:04,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26716.47 MB 2025-02-15 06:10:04,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28606.00 MB 2025-02-15 06:10:04,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:10:04,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30410.80 MB 2025-02-15 06:10:04,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32298.24 MB 2025-02-15 06:10:04,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:10:04,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30023.43 MB 2025-02-15 06:10:04,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:10:04,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:10:04,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:10:04,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28606.00 MB 2025-02-15 06:10:04,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30847.86 MB 2025-02-15 06:10:04,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:10:04,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32298.24 MB 2025-02-15 06:10:04,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38432.41 MB 2025-02-15 06:10:04,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 06:10:04,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36392.14 MB 2025-02-15 06:10:04,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:10:04,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:10:04,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:10:04,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26716.47 MB 2025-02-15 06:10:04,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30847.86 MB 2025-02-15 06:10:04,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:10:04,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30410.80 MB 2025-02-15 06:10:04,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38432.41 MB 2025-02-15 06:10:04,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 06:10:04,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36392.14 MB 2025-02-15 06:10:04,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:10:04,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:10:04,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:10:04,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32381.40 MB 2025-02-15 06:10:04,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33148.40 MB 2025-02-15 06:10:04,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:10:04,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38432.41 MB 2025-02-15 06:10:04,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-15 06:10:04,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:10:04,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33856.19 MB 2025-02-15 06:10:04,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:10:04,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:10:04,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:10:04,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33561.29 MB 2025-02-15 06:10:04,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33789.37 MB 2025-02-15 06:10:04,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-15 06:10:04,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-15 06:10:04,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-15 06:10:04,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:10:04,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33991.11 MB 2025-02-15 06:10:04,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:10:04,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:10:04,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.31 seconds 2025-02-15 06:10:04,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:04,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19943.83 MB 2025-02-15 06:10:04,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33989.50 MB 2025-02-15 06:10:04,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14045.67 MB 2025-02-15 06:10:04,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-15 06:10:04,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-15 06:10:04,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11853.10 MB 2025-02-15 06:10:04,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33991.11 MB 2025-02-15 06:10:05,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:10:05,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:10:05,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:10:05,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:05,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33989.50 MB 2025-02-15 06:10:05,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24934.46 MB 2025-02-15 06:10:05,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9055.04 MB 2025-02-15 06:10:05,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-15 06:10:05,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-15 06:10:05,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:10:05,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36489.50 MB 2025-02-15 06:10:05,052 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 06:10:05,052 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:10:05,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:10:05,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:10:05,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:10:05,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:10:05,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24934.46 MB 2025-02-15 06:10:05,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33335.32 MB 2025-02-15 06:10:05,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 06:10:05,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-15 06:10:05,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47196.41 MB 2025-02-15 06:10:05,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 06:10:05,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33335.32 MB 2025-02-15 06:10:05,222 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 06:10:05,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:10:05,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:10:05,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:10:05,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:10:05,230 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:10:05,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:10:05,232 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:10:05,232 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:11:52,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:11:52,838 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:11:52,847 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:11:52,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:11:52,854 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1566, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:11:52,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:11:52,856 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1566, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:12:16,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:12:16,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:12:16,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.06 seconds 2025-02-15 06:12:16,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:16,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23880.85 MB 2025-02-15 06:12:16,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29423.62 MB 2025-02-15 06:12:16,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5542.77 MB 2025-02-15 06:12:16,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55547.27 MB 2025-02-15 06:12:16,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38713.43 MB 2025-02-15 06:12:16,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16833.84 MB 2025-02-15 06:12:16,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38335.05 MB 2025-02-15 06:12:17,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:12:17,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:12:17,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:12:17,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:17,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29423.62 MB 2025-02-15 06:12:17,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23919.00 MB 2025-02-15 06:12:17,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5504.62 MB 2025-02-15 06:12:17,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38713.43 MB 2025-02-15 06:12:17,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49457.14 MB 2025-02-15 06:12:17,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10743.71 MB 2025-02-15 06:12:17,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45574.45 MB 2025-02-15 06:12:18,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:12:18,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:12:18,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:12:18,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:18,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23919.00 MB 2025-02-15 06:12:18,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24449.84 MB 2025-02-15 06:12:18,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:12:18,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49457.14 MB 2025-02-15 06:12:18,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33170.65 MB 2025-02-15 06:12:18,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16286.48 MB 2025-02-15 06:12:18,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28428.39 MB 2025-02-15 06:12:18,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:12:18,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:12:18,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:12:18,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:18,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24449.84 MB 2025-02-15 06:12:18,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26339.37 MB 2025-02-15 06:12:18,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:12:18,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33170.65 MB 2025-02-15 06:12:18,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33170.65 MB 2025-02-15 06:12:18,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:12:18,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27756.80 MB 2025-02-15 06:12:19,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:12:19,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:12:19,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:12:19,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26339.37 MB 2025-02-15 06:12:19,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28581.23 MB 2025-02-15 06:12:19,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:12:19,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33170.65 MB 2025-02-15 06:12:19,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36001.81 MB 2025-02-15 06:12:19,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:12:19,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34125.51 MB 2025-02-15 06:12:19,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:12:19,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:12:19,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:12:19,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24449.84 MB 2025-02-15 06:12:19,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28581.23 MB 2025-02-15 06:12:19,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:12:19,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33170.65 MB 2025-02-15 06:12:19,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36001.81 MB 2025-02-15 06:12:19,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:12:19,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34125.51 MB 2025-02-15 06:12:19,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:12:19,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:12:19,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:12:19,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30114.77 MB 2025-02-15 06:12:19,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30881.77 MB 2025-02-15 06:12:19,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:12:19,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36001.81 MB 2025-02-15 06:12:19,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36414.95 MB 2025-02-15 06:12:19,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:12:19,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31589.56 MB 2025-02-15 06:12:19,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:12:19,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:12:19,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:12:19,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31294.66 MB 2025-02-15 06:12:19,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31521.70 MB 2025-02-15 06:12:19,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.04 MB 2025-02-15 06:12:19,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36414.95 MB 2025-02-15 06:12:19,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36414.95 MB 2025-02-15 06:12:19,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:12:19,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31744.36 MB 2025-02-15 06:12:19,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:12:19,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:12:19,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.51 seconds 2025-02-15 06:12:19,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18424.78 MB 2025-02-15 06:12:19,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31721.67 MB 2025-02-15 06:12:19,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13296.90 MB 2025-02-15 06:12:19,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55547.27 MB 2025-02-15 06:12:19,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36414.95 MB 2025-02-15 06:12:19,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19132.32 MB 2025-02-15 06:12:19,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31744.36 MB 2025-02-15 06:12:19,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:12:19,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:12:19,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:12:19,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31721.67 MB 2025-02-15 06:12:19,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23412.91 MB 2025-02-15 06:12:19,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8308.76 MB 2025-02-15 06:12:19,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36414.95 MB 2025-02-15 06:12:19,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36414.95 MB 2025-02-15 06:12:19,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:12:19,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34219.51 MB 2025-02-15 06:12:19,651 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 06:12:19,651 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:12:19,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:12:19,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:12:19,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:12:19,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:12:19,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23412.91 MB 2025-02-15 06:12:19,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31805.50 MB 2025-02-15 06:12:19,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-15 06:12:19,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36414.95 MB 2025-02-15 06:12:19,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44759.52 MB 2025-02-15 06:12:19,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-15 06:12:19,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31805.50 MB 2025-02-15 06:12:19,821 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 06:12:19,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:12:19,822 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:12:19,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:12:19,823 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:12:19,828 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:12:19,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:12:19,829 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:12:19,829 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:13:47,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:13:47,010 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:13:47,015 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:13:47,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:13:47,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2114, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:13:47,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:13:47,020 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2114, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:14:19,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:14:19,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:14:19,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.58 seconds 2025-02-15 06:14:19,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:19,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27699.40 MB 2025-02-15 06:14:19,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35180.72 MB 2025-02-15 06:14:19,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7481.33 MB 2025-02-15 06:14:19,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57275.32 MB 2025-02-15 06:14:19,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40647.00 MB 2025-02-15 06:14:19,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16628.32 MB 2025-02-15 06:14:19,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44192.03 MB 2025-02-15 06:14:19,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:14:19,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:14:19,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 06:14:19,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:19,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35180.72 MB 2025-02-15 06:14:19,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26768.93 MB 2025-02-15 06:14:19,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8411.80 MB 2025-02-15 06:14:19,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40647.00 MB 2025-02-15 06:14:19,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56461.62 MB 2025-02-15 06:14:19,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15814.62 MB 2025-02-15 06:14:19,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55771.04 MB 2025-02-15 06:14:21,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:14:21,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:14:21,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:14:21,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:21,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26768.93 MB 2025-02-15 06:14:21,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27299.77 MB 2025-02-15 06:14:21,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:14:21,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56461.62 MB 2025-02-15 06:14:21,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31117.54 MB 2025-02-15 06:14:21,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25344.08 MB 2025-02-15 06:14:21,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31279.94 MB 2025-02-15 06:14:21,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:14:21,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:14:21,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:14:21,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:21,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27299.77 MB 2025-02-15 06:14:21,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29189.30 MB 2025-02-15 06:14:21,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:14:21,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31117.54 MB 2025-02-15 06:14:21,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33004.98 MB 2025-02-15 06:14:21,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:14:21,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30606.73 MB 2025-02-15 06:14:21,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:14:21,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:14:21,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:14:21,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:21,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29189.30 MB 2025-02-15 06:14:21,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31431.16 MB 2025-02-15 06:14:21,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:14:21,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33004.98 MB 2025-02-15 06:14:21,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38667.29 MB 2025-02-15 06:14:21,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:14:21,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.44 MB 2025-02-15 06:14:21,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:14:21,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:14:21,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:14:21,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:21,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27299.77 MB 2025-02-15 06:14:21,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31431.16 MB 2025-02-15 06:14:21,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:14:21,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31117.54 MB 2025-02-15 06:14:21,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38667.29 MB 2025-02-15 06:14:21,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 06:14:21,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.44 MB 2025-02-15 06:14:22,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:14:22,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:14:22,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:14:22,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:22,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32964.70 MB 2025-02-15 06:14:22,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33731.70 MB 2025-02-15 06:14:22,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:14:22,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38667.29 MB 2025-02-15 06:14:22,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39082.52 MB 2025-02-15 06:14:22,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 06:14:22,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34439.49 MB 2025-02-15 06:14:22,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:14:22,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:14:22,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:14:22,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:22,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34144.59 MB 2025-02-15 06:14:22,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34372.96 MB 2025-02-15 06:14:22,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 06:14:22,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39082.52 MB 2025-02-15 06:14:22,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39082.52 MB 2025-02-15 06:14:22,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:14:22,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34594.13 MB 2025-02-15 06:14:22,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:14:22,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:14:22,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.12 seconds 2025-02-15 06:14:22,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:22,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20334.05 MB 2025-02-15 06:14:22,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34573.81 MB 2025-02-15 06:14:22,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14239.76 MB 2025-02-15 06:14:22,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57275.32 MB 2025-02-15 06:14:22,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39082.52 MB 2025-02-15 06:14:22,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18192.79 MB 2025-02-15 06:14:22,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34594.13 MB 2025-02-15 06:14:22,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:14:22,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:14:22,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:14:22,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:22,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34573.81 MB 2025-02-15 06:14:22,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25326.82 MB 2025-02-15 06:14:22,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9247.00 MB 2025-02-15 06:14:22,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39082.52 MB 2025-02-15 06:14:22,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39082.52 MB 2025-02-15 06:14:22,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:14:22,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37075.65 MB 2025-02-15 06:14:22,431 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 06:14:22,432 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:14:22,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:14:22,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:14:22,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:14:22,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:14:22,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25326.82 MB 2025-02-15 06:14:22,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33732.48 MB 2025-02-15 06:14:22,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 06:14:22,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39082.52 MB 2025-02-15 06:14:22,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47441.77 MB 2025-02-15 06:14:22,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 06:14:22,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33732.48 MB 2025-02-15 06:14:22,595 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 06:14:22,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:14:22,597 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:14:22,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:14:22,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:14:22,602 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:14:22,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:14:22,603 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:14:22,603 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:15:18,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:18,521 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:15:18,526 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:15:18,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:18,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1780, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:15:18,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:18,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1780, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:15:46,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:15:46,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:15:46,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.56 seconds 2025-02-15 06:15:46,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:46,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25372.03 MB 2025-02-15 06:15:46,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31671.88 MB 2025-02-15 06:15:46,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6299.84 MB 2025-02-15 06:15:46,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55801.02 MB 2025-02-15 06:15:46,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39474.69 MB 2025-02-15 06:15:46,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16326.33 MB 2025-02-15 06:15:46,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40505.71 MB 2025-02-15 06:15:46,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:15:46,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:15:46,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:15:46,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:46,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31671.88 MB 2025-02-15 06:15:46,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25031.52 MB 2025-02-15 06:15:46,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6640.36 MB 2025-02-15 06:15:46,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39474.69 MB 2025-02-15 06:15:46,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52615.45 MB 2025-02-15 06:15:46,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13140.75 MB 2025-02-15 06:15:46,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49302.97 MB 2025-02-15 06:15:48,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:15:48,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:15:48,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 06:15:48,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25031.52 MB 2025-02-15 06:15:48,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25562.36 MB 2025-02-15 06:15:48,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:15:48,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52615.45 MB 2025-02-15 06:15:48,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-15 06:15:48,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22196.26 MB 2025-02-15 06:15:48,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29540.90 MB 2025-02-15 06:15:48,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:15:48,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:15:48,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:15:48,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-15 06:15:48,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27451.89 MB 2025-02-15 06:15:48,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:15:48,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-15 06:15:48,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-15 06:15:48,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:15:48,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.32 MB 2025-02-15 06:15:48,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:15:48,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:15:48,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:15:48,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27451.89 MB 2025-02-15 06:15:48,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-15 06:15:48,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:15:48,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-15 06:15:48,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37025.22 MB 2025-02-15 06:15:48,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:15:48,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-15 06:15:48,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:15:48,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:15:48,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 06:15:48,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-15 06:15:48,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-15 06:15:48,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:15:48,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-15 06:15:48,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37025.22 MB 2025-02-15 06:15:48,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:15:48,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-15 06:15:48,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:15:48,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:15:48,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:15:48,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31227.29 MB 2025-02-15 06:15:48,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31994.29 MB 2025-02-15 06:15:48,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:15:48,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37025.22 MB 2025-02-15 06:15:48,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37438.36 MB 2025-02-15 06:15:48,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:15:48,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32702.08 MB 2025-02-15 06:15:48,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:15:48,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:15:48,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:15:48,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.18 MB 2025-02-15 06:15:48,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32636.26 MB 2025-02-15 06:15:48,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-15 06:15:48,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37438.36 MB 2025-02-15 06:15:48,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37438.36 MB 2025-02-15 06:15:48,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:15:48,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32850.92 MB 2025-02-15 06:15:48,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:15:48,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:15:48,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.14 seconds 2025-02-15 06:15:48,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19170.37 MB 2025-02-15 06:15:48,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32837.26 MB 2025-02-15 06:15:48,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13666.89 MB 2025-02-15 06:15:48,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55801.02 MB 2025-02-15 06:15:48,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37438.36 MB 2025-02-15 06:15:48,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18362.66 MB 2025-02-15 06:15:48,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32850.92 MB 2025-02-15 06:15:48,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:15:48,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:15:48,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:15:48,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32837.26 MB 2025-02-15 06:15:48,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24173.62 MB 2025-02-15 06:15:48,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8663.65 MB 2025-02-15 06:15:48,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37438.36 MB 2025-02-15 06:15:48,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37438.36 MB 2025-02-15 06:15:48,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:15:48,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35348.01 MB 2025-02-15 06:15:48,963 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 06:15:48,963 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:15:48,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:15:48,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:15:48,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:15:48,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:15:48,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24173.62 MB 2025-02-15 06:15:48,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32609.21 MB 2025-02-15 06:15:48,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 06:15:48,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37438.36 MB 2025-02-15 06:15:48,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45826.97 MB 2025-02-15 06:15:48,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 06:15:48,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32609.21 MB 2025-02-15 06:15:49,131 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 06:15:49,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:49,132 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:15:49,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:49,133 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:15:49,138 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:15:49,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:15:49,139 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:15:49,139 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:17:34,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:34,392 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:17:34,397 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:17:34,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:34,401 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1255, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:17:34,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:34,402 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1255, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:17:53,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:17:53,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:17:53,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.22 seconds 2025-02-15 06:17:53,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:53,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21713.75 MB 2025-02-15 06:17:53,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26155.51 MB 2025-02-15 06:17:53,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4441.77 MB 2025-02-15 06:17:53,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54215.57 MB 2025-02-15 06:17:53,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37639.68 MB 2025-02-15 06:17:53,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16575.89 MB 2025-02-15 06:17:53,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35035.49 MB 2025-02-15 06:17:53,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:17:53,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:17:53,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:17:53,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:53,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.51 MB 2025-02-15 06:17:53,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22302.20 MB 2025-02-15 06:17:53,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3853.31 MB 2025-02-15 06:17:53,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37639.68 MB 2025-02-15 06:17:53,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44207.96 MB 2025-02-15 06:17:53,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6568.28 MB 2025-02-15 06:17:53,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39223.65 MB 2025-02-15 06:17:55,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:17:55,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:17:55,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:17:55,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:55,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22302.20 MB 2025-02-15 06:17:55,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22833.05 MB 2025-02-15 06:17:55,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:17:55,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44207.96 MB 2025-02-15 06:17:55,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 06:17:55,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15181.28 MB 2025-02-15 06:17:55,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26811.59 MB 2025-02-15 06:17:55,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:17:55,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:17:55,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:17:55,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:55,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-15 06:17:55,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24722.58 MB 2025-02-15 06:17:55,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:17:55,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 06:17:55,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 06:17:55,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:17:55,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26140.01 MB 2025-02-15 06:17:55,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:17:55,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:17:55,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:17:55,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:55,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24722.58 MB 2025-02-15 06:17:55,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-15 06:17:55,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:17:55,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 06:17:55,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34688.99 MB 2025-02-15 06:17:55,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:17:55,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-15 06:17:55,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:17:55,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:17:55,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:17:55,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:55,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-15 06:17:55,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-15 06:17:55,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:17:55,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 06:17:55,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34688.99 MB 2025-02-15 06:17:55,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:17:55,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-15 06:17:56,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:17:56,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:17:56,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:17:56,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:56,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28497.98 MB 2025-02-15 06:17:56,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29264.98 MB 2025-02-15 06:17:56,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:17:56,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34688.99 MB 2025-02-15 06:17:56,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-15 06:17:56,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:17:56,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29972.77 MB 2025-02-15 06:17:56,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:17:56,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:17:56,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:17:56,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:56,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29677.87 MB 2025-02-15 06:17:56,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29906.64 MB 2025-02-15 06:17:56,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.78 MB 2025-02-15 06:17:56,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-15 06:17:56,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-15 06:17:56,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:17:56,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30139.72 MB 2025-02-15 06:17:56,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:17:56,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:17:56,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.63 seconds 2025-02-15 06:17:56,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:56,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17341.23 MB 2025-02-15 06:17:56,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30106.51 MB 2025-02-15 06:17:56,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12765.29 MB 2025-02-15 06:17:56,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54215.57 MB 2025-02-15 06:17:56,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-15 06:17:56,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19113.44 MB 2025-02-15 06:17:56,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30139.72 MB 2025-02-15 06:17:56,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:17:56,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:17:56,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:17:56,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:56,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30106.51 MB 2025-02-15 06:17:56,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22327.93 MB 2025-02-15 06:17:56,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7778.58 MB 2025-02-15 06:17:56,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-15 06:17:56,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-15 06:17:56,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:17:56,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.13 MB 2025-02-15 06:17:56,342 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 06:17:56,343 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:17:56,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:17:56,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:17:56,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 06:17:56,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:17:56,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22327.93 MB 2025-02-15 06:17:56,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30716.35 MB 2025-02-15 06:17:56,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 06:17:56,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-15 06:17:56,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43442.50 MB 2025-02-15 06:17:56,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-15 06:17:56,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30716.35 MB 2025-02-15 06:17:56,599 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 06:17:56,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:56,602 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:17:56,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:56,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:17:56,611 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:17:56,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:17:56,613 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:17:56,613 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:18:14,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:14,462 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:18:14,470 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:18:14,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:14,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2375, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:18:14,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:14,479 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2375, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:18:51,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:18:51,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:18:51,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.06 seconds 2025-02-15 06:18:51,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:51,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29518.48 MB 2025-02-15 06:18:51,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37923.86 MB 2025-02-15 06:18:51,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.39 MB 2025-02-15 06:18:51,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64357.40 MB 2025-02-15 06:18:51,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41630.56 MB 2025-02-15 06:18:51,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22726.84 MB 2025-02-15 06:18:51,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46917.08 MB 2025-02-15 06:18:51,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:18:51,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:18:51,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:18:51,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:51,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37923.86 MB 2025-02-15 06:18:51,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28125.98 MB 2025-02-15 06:18:51,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9797.88 MB 2025-02-15 06:18:51,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41630.56 MB 2025-02-15 06:18:51,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59462.65 MB 2025-02-15 06:18:51,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17832.08 MB 2025-02-15 06:18:51,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61544.04 MB 2025-02-15 06:18:53,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:18:53,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:18:53,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 06:18:53,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:53,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28125.98 MB 2025-02-15 06:18:53,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28656.82 MB 2025-02-15 06:18:53,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:18:53,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59462.65 MB 2025-02-15 06:18:53,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31146.90 MB 2025-02-15 06:18:53,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28315.75 MB 2025-02-15 06:18:53,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32635.37 MB 2025-02-15 06:18:53,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:18:53,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:18:53,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:18:53,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:53,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28656.82 MB 2025-02-15 06:18:53,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30546.36 MB 2025-02-15 06:18:53,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:18:53,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31146.90 MB 2025-02-15 06:18:53,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33978.06 MB 2025-02-15 06:18:53,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:18:53,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.78 MB 2025-02-15 06:18:53,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:18:53,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:18:53,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:18:53,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:53,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30546.36 MB 2025-02-15 06:18:53,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32788.21 MB 2025-02-15 06:18:53,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:18:53,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33978.06 MB 2025-02-15 06:18:53,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40112.23 MB 2025-02-15 06:18:53,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 06:18:53,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38332.49 MB 2025-02-15 06:18:53,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:18:53,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:18:53,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:18:53,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:53,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28656.82 MB 2025-02-15 06:18:53,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32788.21 MB 2025-02-15 06:18:53,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:18:53,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31146.90 MB 2025-02-15 06:18:53,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40112.23 MB 2025-02-15 06:18:53,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 06:18:53,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38332.49 MB 2025-02-15 06:18:54,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:18:54,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:18:54,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:18:54,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:54,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34321.75 MB 2025-02-15 06:18:54,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35088.76 MB 2025-02-15 06:18:54,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:18:54,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40112.23 MB 2025-02-15 06:18:54,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 06:18:54,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:18:54,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35796.54 MB 2025-02-15 06:18:54,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:18:54,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:18:54,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:18:54,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:54,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35501.64 MB 2025-02-15 06:18:54,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35730.19 MB 2025-02-15 06:18:54,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-15 06:18:54,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-15 06:18:54,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 06:18:54,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:18:54,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35938.87 MB 2025-02-15 06:18:54,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:18:54,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:18:54,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.66 seconds 2025-02-15 06:18:54,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:54,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21243.59 MB 2025-02-15 06:18:54,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35930.65 MB 2025-02-15 06:18:54,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14687.05 MB 2025-02-15 06:18:54,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60154.71 MB 2025-02-15 06:18:54,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 06:18:54,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19629.34 MB 2025-02-15 06:18:54,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35938.87 MB 2025-02-15 06:18:54,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:18:54,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:18:54,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:18:54,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:54,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35930.65 MB 2025-02-15 06:18:54,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26238.85 MB 2025-02-15 06:18:54,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9691.79 MB 2025-02-15 06:18:54,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-15 06:18:54,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 06:18:54,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:18:54,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38434.63 MB 2025-02-15 06:18:54,433 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 06:18:54,433 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:18:54,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:18:54,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:18:54,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:18:54,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:18:54,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26238.85 MB 2025-02-15 06:18:54,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34652.38 MB 2025-02-15 06:18:54,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-15 06:18:54,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-15 06:18:54,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48888.81 MB 2025-02-15 06:18:54,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 06:18:54,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34652.38 MB 2025-02-15 06:18:54,603 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 06:18:54,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:54,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:18:54,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:54,605 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:18:54,610 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:18:54,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:18:54,611 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:18:54,611 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:20:20,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:20,190 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:20:20,195 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:20:20,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:20,199 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:20:20,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:20,200 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:20:24,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:20:24,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:20:24,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.39 seconds 2025-02-15 06:20:24,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:24,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-15 06:20:24,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.22 MB 2025-02-15 06:20:24,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.52 MB 2025-02-15 06:20:24,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57252.25 MB 2025-02-15 06:20:24,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20715.67 MB 2025-02-15 06:20:24,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36536.58 MB 2025-02-15 06:20:24,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24865.05 MB 2025-02-15 06:20:24,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:20:24,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:20:24,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:20:24,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:24,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.22 MB 2025-02-15 06:20:24,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.80 MB 2025-02-15 06:20:24,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 0.58 MB 2025-02-15 06:20:24,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20715.67 MB 2025-02-15 06:20:24,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20715.67 MB 2025-02-15 06:20:24,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:20:24,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18948.07 MB 2025-02-15 06:20:25,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:20:25,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:20:25,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-15 06:20:25,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:25,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.80 MB 2025-02-15 06:20:25,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16226.80 MB 2025-02-15 06:20:25,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.00 MB 2025-02-15 06:20:25,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20715.67 MB 2025-02-15 06:20:25,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20174.60 MB 2025-02-15 06:20:25,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -541.07 MB 2025-02-15 06:20:25,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20198.42 MB 2025-02-15 06:20:25,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:20:25,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:20:25,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:20:25,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:25,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16226.80 MB 2025-02-15 06:20:25,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17237.98 MB 2025-02-15 06:20:25,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1011.18 MB 2025-02-15 06:20:25,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20174.60 MB 2025-02-15 06:20:25,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20680.02 MB 2025-02-15 06:20:25,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 505.41 MB 2025-02-15 06:20:25,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17996.30 MB 2025-02-15 06:20:25,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:20:25,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:20:25,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 06:20:25,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:25,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17237.98 MB 2025-02-15 06:20:25,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18437.40 MB 2025-02-15 06:20:25,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1199.42 MB 2025-02-15 06:20:25,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20680.02 MB 2025-02-15 06:20:25,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23460.84 MB 2025-02-15 06:20:25,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2780.82 MB 2025-02-15 06:20:25,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21405.66 MB 2025-02-15 06:20:25,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:20:25,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:20:25,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:20:25,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:25,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16226.80 MB 2025-02-15 06:20:25,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18437.40 MB 2025-02-15 06:20:25,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2210.60 MB 2025-02-15 06:20:25,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20174.60 MB 2025-02-15 06:20:25,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23460.84 MB 2025-02-15 06:20:25,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3286.24 MB 2025-02-15 06:20:25,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21405.66 MB 2025-02-15 06:20:26,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:20:26,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:20:26,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 06:20:26,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:26,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19257.84 MB 2025-02-15 06:20:26,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.19 MB 2025-02-15 06:20:26,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.35 MB 2025-02-15 06:20:26,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23460.84 MB 2025-02-15 06:20:26,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23678.94 MB 2025-02-15 06:20:26,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 218.10 MB 2025-02-15 06:20:26,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20047.48 MB 2025-02-15 06:20:26,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:20:26,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:20:26,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:20:26,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:26,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19889.09 MB 2025-02-15 06:20:26,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20109.62 MB 2025-02-15 06:20:26,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.53 MB 2025-02-15 06:20:26,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23678.94 MB 2025-02-15 06:20:26,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23678.94 MB 2025-02-15 06:20:26,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:20:26,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20143.63 MB 2025-02-15 06:20:26,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:20:26,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:20:26,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.84 seconds 2025-02-15 06:20:26,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:26,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13954.70 MB 2025-02-15 06:20:26,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20310.15 MB 2025-02-15 06:20:26,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6355.45 MB 2025-02-15 06:20:26,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57252.25 MB 2025-02-15 06:20:26,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23678.94 MB 2025-02-15 06:20:26,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33573.31 MB 2025-02-15 06:20:26,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.15 MB 2025-02-15 06:20:26,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:20:26,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:20:26,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 06:20:26,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:26,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15067.00 MB 2025-02-15 06:20:26,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18072.92 MB 2025-02-15 06:20:26,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.92 MB 2025-02-15 06:20:26,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23678.94 MB 2025-02-15 06:20:26,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23678.94 MB 2025-02-15 06:20:26,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:20:26,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18373.48 MB 2025-02-15 06:20:26,354 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 06:20:26,354 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:20:26,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:20:26,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:20:26,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:20:26,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:20:26,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18072.92 MB 2025-02-15 06:20:26,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26489.52 MB 2025-02-15 06:20:26,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 06:20:26,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23678.94 MB 2025-02-15 06:20:26,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34139.54 MB 2025-02-15 06:20:26,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-15 06:20:26,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26489.52 MB 2025-02-15 06:20:26,530 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 06:20:26,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:26,531 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:20:26,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:26,532 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:20:26,537 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:20:26,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:20:26,538 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:20:26,538 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:22:12,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:12,002 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:22:12,011 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:22:12,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:12,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:22:12,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:12,020 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:22:38,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:22:38,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:22:38,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.75 seconds 2025-02-15 06:22:38,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:38,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.24 MB 2025-02-15 06:22:38,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31272.87 MB 2025-02-15 06:22:38,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6165.63 MB 2025-02-15 06:22:38,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42507.17 MB 2025-02-15 06:22:38,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39378.22 MB 2025-02-15 06:22:38,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3128.95 MB 2025-02-15 06:22:38,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40240.92 MB 2025-02-15 06:22:38,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:22:38,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:22:38,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 06:22:38,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:38,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31272.87 MB 2025-02-15 06:22:38,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24833.97 MB 2025-02-15 06:22:38,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6438.90 MB 2025-02-15 06:22:38,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39378.22 MB 2025-02-15 06:22:38,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52294.58 MB 2025-02-15 06:22:38,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12916.36 MB 2025-02-15 06:22:38,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48564.04 MB 2025-02-15 06:22:40,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:22:40,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:22:40,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:22:40,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:40,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24833.97 MB 2025-02-15 06:22:40,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25364.81 MB 2025-02-15 06:22:40,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:22:40,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52294.58 MB 2025-02-15 06:22:40,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-15 06:22:40,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17666.41 MB 2025-02-15 06:22:40,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.35 MB 2025-02-15 06:22:40,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:22:40,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:22:40,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:22:40,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:40,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-15 06:22:40,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27254.34 MB 2025-02-15 06:22:40,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:22:40,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-15 06:22:40,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-15 06:22:40,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:22:40,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28671.77 MB 2025-02-15 06:22:41,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:22:41,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:22:41,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:22:41,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27254.34 MB 2025-02-15 06:22:41,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-15 06:22:41,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:22:41,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-15 06:22:41,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37459.33 MB 2025-02-15 06:22:41,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:22:41,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-15 06:22:41,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:22:41,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:22:41,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:22:41,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-15 06:22:41,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-15 06:22:41,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:22:41,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-15 06:22:41,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37459.33 MB 2025-02-15 06:22:41,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:22:41,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-15 06:22:41,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:22:41,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:22:41,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:22:41,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31029.74 MB 2025-02-15 06:22:41,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31796.74 MB 2025-02-15 06:22:41,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:22:41,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37459.33 MB 2025-02-15 06:22:41,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-15 06:22:41,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:22:41,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32504.53 MB 2025-02-15 06:22:41,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:22:41,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:22:41,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:22:41,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.63 MB 2025-02-15 06:22:41,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32438.69 MB 2025-02-15 06:22:41,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 06:22:41,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-15 06:22:41,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-15 06:22:41,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:22:41,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32644.13 MB 2025-02-15 06:22:41,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:22:41,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:22:41,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.20 seconds 2025-02-15 06:22:41,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.97 MB 2025-02-15 06:22:41,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32639.66 MB 2025-02-15 06:22:41,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13601.69 MB 2025-02-15 06:22:41,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42507.17 MB 2025-02-15 06:22:41,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-15 06:22:41,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4630.51 MB 2025-02-15 06:22:41,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32644.13 MB 2025-02-15 06:22:41,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:22:41,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:22:41,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:22:41,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32639.66 MB 2025-02-15 06:22:41,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24040.84 MB 2025-02-15 06:22:41,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8598.82 MB 2025-02-15 06:22:41,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-15 06:22:41,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-15 06:22:41,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:22:41,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35150.10 MB 2025-02-15 06:22:41,514 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 06:22:41,515 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:22:41,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:22:41,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:22:41,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:22:41,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:22:41,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24040.84 MB 2025-02-15 06:22:41,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32475.69 MB 2025-02-15 06:22:41,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 06:22:41,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-15 06:22:41,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46263.17 MB 2025-02-15 06:22:41,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 06:22:41,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32475.69 MB 2025-02-15 06:22:41,679 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 06:22:41,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:41,681 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:22:41,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:41,682 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:22:41,686 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:22:41,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:22:41,687 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:22:41,688 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:23:02,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:02,912 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:23:02,916 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:23:02,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:02,920 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:23:02,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:02,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:23:39,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:23:39,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:23:39,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.23 seconds 2025-02-15 06:23:39,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:39,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.46 MB 2025-02-15 06:23:39,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37471.27 MB 2025-02-15 06:23:39,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8252.82 MB 2025-02-15 06:23:39,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58841.89 MB 2025-02-15 06:23:39,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41466.99 MB 2025-02-15 06:23:39,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17374.90 MB 2025-02-15 06:23:39,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46390.57 MB 2025-02-15 06:23:39,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:23:39,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:23:39,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:23:39,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:39,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37471.27 MB 2025-02-15 06:23:39,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27902.24 MB 2025-02-15 06:23:39,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9569.03 MB 2025-02-15 06:23:39,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41466.99 MB 2025-02-15 06:23:39,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58728.64 MB 2025-02-15 06:23:39,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17261.66 MB 2025-02-15 06:23:39,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60239.65 MB 2025-02-15 06:23:41,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:23:41,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:23:41,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:23:41,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27902.24 MB 2025-02-15 06:23:41,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28433.08 MB 2025-02-15 06:23:41,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:23:41,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58728.64 MB 2025-02-15 06:23:41,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31153.19 MB 2025-02-15 06:23:41,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27575.45 MB 2025-02-15 06:23:41,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32412.41 MB 2025-02-15 06:23:41,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:23:41,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:23:41,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:23:41,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28433.08 MB 2025-02-15 06:23:41,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.62 MB 2025-02-15 06:23:41,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:23:41,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31153.19 MB 2025-02-15 06:23:41,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-15 06:23:41,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:23:41,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31740.04 MB 2025-02-15 06:23:41,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:23:41,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:23:41,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:23:41,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30322.62 MB 2025-02-15 06:23:41,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32564.47 MB 2025-02-15 06:23:41,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:23:41,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33984.35 MB 2025-02-15 06:23:41,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40118.52 MB 2025-02-15 06:23:41,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 06:23:41,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38108.75 MB 2025-02-15 06:23:41,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:23:41,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:23:41,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:23:41,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28433.08 MB 2025-02-15 06:23:41,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32564.47 MB 2025-02-15 06:23:41,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:23:41,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31153.19 MB 2025-02-15 06:23:41,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40118.52 MB 2025-02-15 06:23:41,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 06:23:41,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38108.75 MB 2025-02-15 06:23:41,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:23:41,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:23:41,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 06:23:41,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34098.01 MB 2025-02-15 06:23:41,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34865.02 MB 2025-02-15 06:23:41,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:23:41,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40118.52 MB 2025-02-15 06:23:41,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 06:23:41,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:23:41,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35572.80 MB 2025-02-15 06:23:41,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:23:41,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:23:41,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:23:41,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35277.90 MB 2025-02-15 06:23:41,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35507.38 MB 2025-02-15 06:23:41,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.48 MB 2025-02-15 06:23:41,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 06:23:41,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 06:23:41,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:23:41,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35723.29 MB 2025-02-15 06:23:41,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:23:41,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:23:41,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.83 seconds 2025-02-15 06:23:41,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:41,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21093.58 MB 2025-02-15 06:23:41,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35708.45 MB 2025-02-15 06:23:41,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14614.87 MB 2025-02-15 06:23:41,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58841.89 MB 2025-02-15 06:23:41,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 06:23:41,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18306.04 MB 2025-02-15 06:23:41,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35723.29 MB 2025-02-15 06:23:42,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:23:42,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:23:42,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:23:42,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:42,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35708.45 MB 2025-02-15 06:23:42,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26097.97 MB 2025-02-15 06:23:42,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9610.48 MB 2025-02-15 06:23:42,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 06:23:42,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 06:23:42,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:23:42,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38220.12 MB 2025-02-15 06:23:42,037 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:23:42,038 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:23:42,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:23:42,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:23:42,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:23:42,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:42,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26097.97 MB 2025-02-15 06:23:42,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34536.99 MB 2025-02-15 06:23:42,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:23:42,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 06:23:42,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48926.56 MB 2025-02-15 06:23:42,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:23:42,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34536.99 MB 2025-02-15 06:23:42,203 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:23:42,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:42,204 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:23:42,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:42,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:23:42,210 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:23:42,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:42,211 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:23:42,211 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:23:51,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:51,061 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:23:51,066 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:23:51,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:51,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 409, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:23:51,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:23:51,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 409, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:23:57,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:23:57,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:23:57,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.41 seconds 2025-02-15 06:23:57,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:57,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15818.68 MB 2025-02-15 06:23:57,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17266.11 MB 2025-02-15 06:23:57,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1447.43 MB 2025-02-15 06:23:57,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61511.57 MB 2025-02-15 06:23:57,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19073.60 MB 2025-02-15 06:23:57,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42437.97 MB 2025-02-15 06:23:57,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26196.02 MB 2025-02-15 06:23:57,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:23:57,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:23:57,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 06:23:57,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:57,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17266.11 MB 2025-02-15 06:23:57,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17806.45 MB 2025-02-15 06:23:57,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 540.34 MB 2025-02-15 06:23:57,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19073.60 MB 2025-02-15 06:23:57,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23838.33 MB 2025-02-15 06:23:57,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4764.73 MB 2025-02-15 06:23:57,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22690.06 MB 2025-02-15 06:23:59,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:23:59,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:23:59,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.88 seconds 2025-02-15 06:23:59,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17806.45 MB 2025-02-15 06:23:59,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18318.71 MB 2025-02-15 06:23:59,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-15 06:23:59,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23838.33 MB 2025-02-15 06:23:59,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20180.89 MB 2025-02-15 06:23:59,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3657.43 MB 2025-02-15 06:23:59,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22316.87 MB 2025-02-15 06:23:59,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:23:59,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:23:59,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:23:59,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.71 MB 2025-02-15 06:23:59,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20142.18 MB 2025-02-15 06:23:59,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.47 MB 2025-02-15 06:23:59,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20180.89 MB 2025-02-15 06:23:59,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22917.68 MB 2025-02-15 06:23:59,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2736.78 MB 2025-02-15 06:23:59,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21510.00 MB 2025-02-15 06:23:59,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:23:59,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:23:59,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:23:59,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20142.18 MB 2025-02-15 06:23:59,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.58 MB 2025-02-15 06:23:59,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2163.39 MB 2025-02-15 06:23:59,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22917.68 MB 2025-02-15 06:23:59,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29305.60 MB 2025-02-15 06:23:59,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6387.92 MB 2025-02-15 06:23:59,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27655.81 MB 2025-02-15 06:23:59,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:23:59,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:23:59,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:23:59,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.71 MB 2025-02-15 06:23:59,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.58 MB 2025-02-15 06:23:59,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3986.87 MB 2025-02-15 06:23:59,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20180.89 MB 2025-02-15 06:23:59,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29305.60 MB 2025-02-15 06:23:59,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9124.71 MB 2025-02-15 06:23:59,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27655.81 MB 2025-02-15 06:23:59,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:23:59,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:23:59,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 06:23:59,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23785.45 MB 2025-02-15 06:23:59,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.60 MB 2025-02-15 06:23:59,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-15 06:23:59,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29305.60 MB 2025-02-15 06:23:59,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29708.26 MB 2025-02-15 06:23:59,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 06:23:59,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25208.62 MB 2025-02-15 06:23:59,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:23:59,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:23:59,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:23:59,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24924.04 MB 2025-02-15 06:23:59,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25131.38 MB 2025-02-15 06:23:59,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.34 MB 2025-02-15 06:23:59,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29708.26 MB 2025-02-15 06:23:59,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29712.45 MB 2025-02-15 06:23:59,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 06:23:59,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25299.90 MB 2025-02-15 06:23:59,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:23:59,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:23:59,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.72 seconds 2025-02-15 06:23:59,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:23:59,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14393.70 MB 2025-02-15 06:23:59,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25332.46 MB 2025-02-15 06:23:59,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10938.76 MB 2025-02-15 06:23:59,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61511.57 MB 2025-02-15 06:23:59,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29712.45 MB 2025-02-15 06:23:59,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31799.12 MB 2025-02-15 06:23:59,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25332.46 MB 2025-02-15 06:24:00,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:24:00,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:24:00,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:24:00,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:00,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25332.46 MB 2025-02-15 06:24:00,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19331.49 MB 2025-02-15 06:24:00,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6000.97 MB 2025-02-15 06:24:00,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29712.45 MB 2025-02-15 06:24:00,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29712.45 MB 2025-02-15 06:24:00,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:24:00,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28045.06 MB 2025-02-15 06:24:00,080 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:24:00,081 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:24:00,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:24:00,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:24:00,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:24:00,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:00,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19331.49 MB 2025-02-15 06:24:00,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27770.51 MB 2025-02-15 06:24:00,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:24:00,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29712.45 MB 2025-02-15 06:24:00,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38103.15 MB 2025-02-15 06:24:00,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:24:00,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27770.51 MB 2025-02-15 06:24:00,246 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:24:00,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:00,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:24:00,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:00,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:24:00,253 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:24:00,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:00,255 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:24:00,255 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:24:50,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:50,551 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:24:50,558 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:24:50,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:50,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:24:50,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:50,567 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:24:52,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:24:52,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:24:52,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.34 seconds 2025-02-15 06:24:52,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:52,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-15 06:24:52,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-15 06:24:52,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-15 06:24:52,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50688.16 MB 2025-02-15 06:24:52,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18077.45 MB 2025-02-15 06:24:52,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32610.71 MB 2025-02-15 06:24:52,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.46 MB 2025-02-15 06:24:52,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:24:52,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:24:52,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:24:52,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:52,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-15 06:24:52,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14740.86 MB 2025-02-15 06:24:52,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.62 MB 2025-02-15 06:24:52,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18077.45 MB 2025-02-15 06:24:52,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18077.45 MB 2025-02-15 06:24:52,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:24:52,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16528.98 MB 2025-02-15 06:24:53,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:24:53,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:24:53,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.73 seconds 2025-02-15 06:24:53,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14740.86 MB 2025-02-15 06:24:53,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14933.29 MB 2025-02-15 06:24:53,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 06:24:53,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18077.45 MB 2025-02-15 06:24:53,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18077.45 MB 2025-02-15 06:24:53,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:24:53,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18911.54 MB 2025-02-15 06:24:53,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:24:53,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:24:53,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:24:53,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-15 06:24:53,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15618.01 MB 2025-02-15 06:24:53,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 06:24:53,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18077.45 MB 2025-02-15 06:24:53,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18077.45 MB 2025-02-15 06:24:53,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:24:53,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16131.83 MB 2025-02-15 06:24:53,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:24:53,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:24:53,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:24:53,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15618.01 MB 2025-02-15 06:24:53,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-15 06:24:53,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 06:24:53,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18077.45 MB 2025-02-15 06:24:53,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19453.18 MB 2025-02-15 06:24:53,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 06:24:53,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-15 06:24:53,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:24:53,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:24:53,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:24:53,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-15 06:24:53,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-15 06:24:53,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 06:24:53,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18077.45 MB 2025-02-15 06:24:53,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19453.18 MB 2025-02-15 06:24:53,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 06:24:53,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-15 06:24:53,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:24:53,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:24:53,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:24:53,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16986.63 MB 2025-02-15 06:24:53,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17264.67 MB 2025-02-15 06:24:53,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 06:24:53,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19453.18 MB 2025-02-15 06:24:53,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19599.98 MB 2025-02-15 06:24:53,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-15 06:24:53,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17530.29 MB 2025-02-15 06:24:53,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:24:53,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:24:53,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:24:53,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17414.35 MB 2025-02-15 06:24:53,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17643.33 MB 2025-02-15 06:24:53,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.98 MB 2025-02-15 06:24:53,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19599.98 MB 2025-02-15 06:24:53,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19599.98 MB 2025-02-15 06:24:53,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:24:53,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17650.90 MB 2025-02-15 06:24:53,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:24:53,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:24:53,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.33 seconds 2025-02-15 06:24:53,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:53,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-15 06:24:53,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17844.13 MB 2025-02-15 06:24:53,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4370.23 MB 2025-02-15 06:24:53,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50688.16 MB 2025-02-15 06:24:53,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19599.98 MB 2025-02-15 06:24:53,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31088.18 MB 2025-02-15 06:24:53,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17844.13 MB 2025-02-15 06:24:54,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:24:54,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:24:54,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 06:24:54,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:54,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.13 MB 2025-02-15 06:24:54,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17270.68 MB 2025-02-15 06:24:54,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -573.45 MB 2025-02-15 06:24:54,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19599.98 MB 2025-02-15 06:24:54,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-15 06:24:54,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 06:24:54,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18948.22 MB 2025-02-15 06:24:54,217 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 06:24:54,217 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:24:54,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:24:54,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:24:54,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:24:54,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:24:54,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17270.68 MB 2025-02-15 06:24:54,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25698.01 MB 2025-02-15 06:24:54,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 06:24:54,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-15 06:24:54,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30477.91 MB 2025-02-15 06:24:54,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 06:24:54,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25698.01 MB 2025-02-15 06:24:54,479 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 06:24:54,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:54,482 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:24:54,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:54,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:24:54,491 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:24:54,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:24:54,493 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:24:54,493 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:25:48,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:25:48,503 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:25:48,510 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:25:48,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:25:48,517 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1035, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:25:48,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:25:48,519 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1035, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:26:04,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:26:04,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:26:04,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.99 seconds 2025-02-15 06:26:04,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:04,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20180.75 MB 2025-02-15 06:26:04,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23844.48 MB 2025-02-15 06:26:04,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3663.72 MB 2025-02-15 06:26:04,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38858.13 MB 2025-02-15 06:26:04,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26428.31 MB 2025-02-15 06:26:04,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12429.82 MB 2025-02-15 06:26:04,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32823.82 MB 2025-02-15 06:26:04,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:26:04,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:26:04,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:26:04,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:04,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23844.48 MB 2025-02-15 06:26:04,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21159.54 MB 2025-02-15 06:26:04,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2684.94 MB 2025-02-15 06:26:04,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26428.31 MB 2025-02-15 06:26:04,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35292.97 MB 2025-02-15 06:26:04,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8864.66 MB 2025-02-15 06:26:04,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34805.39 MB 2025-02-15 06:26:06,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:26:06,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:26:06,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:26:06,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21159.54 MB 2025-02-15 06:26:06,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21690.38 MB 2025-02-15 06:26:06,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:26:06,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35292.97 MB 2025-02-15 06:26:06,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24889.00 MB 2025-02-15 06:26:06,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10403.97 MB 2025-02-15 06:26:06,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25669.97 MB 2025-02-15 06:26:06,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:26:06,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:26:06,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:26:06,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21690.38 MB 2025-02-15 06:26:06,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23579.92 MB 2025-02-15 06:26:06,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:26:06,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24889.00 MB 2025-02-15 06:26:06,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26776.44 MB 2025-02-15 06:26:06,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:26:06,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24997.35 MB 2025-02-15 06:26:06,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:26:06,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:26:06,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:26:06,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23579.92 MB 2025-02-15 06:26:06,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25822.82 MB 2025-02-15 06:26:06,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 06:26:06,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26776.44 MB 2025-02-15 06:26:06,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33384.56 MB 2025-02-15 06:26:06,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 06:26:06,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31367.10 MB 2025-02-15 06:26:06,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:26:06,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:26:06,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:26:06,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21690.38 MB 2025-02-15 06:26:06,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25822.82 MB 2025-02-15 06:26:06,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 06:26:06,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24889.00 MB 2025-02-15 06:26:06,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33384.56 MB 2025-02-15 06:26:06,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-15 06:26:06,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31367.10 MB 2025-02-15 06:26:06,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:26:06,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:26:06,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:26:06,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27356.36 MB 2025-02-15 06:26:06,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28123.37 MB 2025-02-15 06:26:06,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:26:06,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33384.56 MB 2025-02-15 06:26:06,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33799.80 MB 2025-02-15 06:26:06,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 06:26:06,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28831.15 MB 2025-02-15 06:26:06,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:26:06,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:26:06,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:26:06,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28536.25 MB 2025-02-15 06:26:06,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28766.10 MB 2025-02-15 06:26:06,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.85 MB 2025-02-15 06:26:06,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33799.80 MB 2025-02-15 06:26:06,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33799.80 MB 2025-02-15 06:26:06,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:26:06,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28976.45 MB 2025-02-15 06:26:06,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:26:06,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:26:06,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.42 seconds 2025-02-15 06:26:06,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:06,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16574.73 MB 2025-02-15 06:26:06,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28967.17 MB 2025-02-15 06:26:06,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12392.44 MB 2025-02-15 06:26:06,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38858.13 MB 2025-02-15 06:26:06,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33799.80 MB 2025-02-15 06:26:06,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5058.33 MB 2025-02-15 06:26:06,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28976.45 MB 2025-02-15 06:26:07,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:26:07,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:26:07,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:26:07,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:07,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28967.17 MB 2025-02-15 06:26:07,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21579.12 MB 2025-02-15 06:26:07,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7388.06 MB 2025-02-15 06:26:07,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33799.80 MB 2025-02-15 06:26:07,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33799.80 MB 2025-02-15 06:26:07,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:26:07,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31478.84 MB 2025-02-15 06:26:07,230 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:26:07,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:26:07,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:26:07,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:26:07,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:26:07,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:26:07,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21579.12 MB 2025-02-15 06:26:07,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30018.14 MB 2025-02-15 06:26:07,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:26:07,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33799.80 MB 2025-02-15 06:26:07,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42190.50 MB 2025-02-15 06:26:07,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:26:07,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30018.14 MB 2025-02-15 06:26:07,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:26:07,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:07,396 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:26:07,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:07,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:26:07,402 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:26:07,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:07,403 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:26:07,403 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:26:52,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:52,852 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:26:52,857 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:26:52,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:52,860 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1295, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:26:52,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:26:52,861 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1295, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:27:12,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:27:12,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:27:12,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.95 seconds 2025-02-15 06:27:12,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:12,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21992.47 MB 2025-02-15 06:27:12,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26575.41 MB 2025-02-15 06:27:12,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4582.93 MB 2025-02-15 06:27:12,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54775.51 MB 2025-02-15 06:27:12,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37836.82 MB 2025-02-15 06:27:12,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16938.70 MB 2025-02-15 06:27:12,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35540.71 MB 2025-02-15 06:27:12,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:27:12,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:27:12,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:27:12,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:12,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26575.41 MB 2025-02-15 06:27:12,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22510.15 MB 2025-02-15 06:27:12,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4065.25 MB 2025-02-15 06:27:12,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37836.82 MB 2025-02-15 06:27:12,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46877.64 MB 2025-02-15 06:27:12,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9040.82 MB 2025-02-15 06:27:12,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40169.29 MB 2025-02-15 06:27:14,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:27:14,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:27:14,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:27:14,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:14,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22510.15 MB 2025-02-15 06:27:14,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23040.99 MB 2025-02-15 06:27:14,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:27:14,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46877.64 MB 2025-02-15 06:27:14,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29058.14 MB 2025-02-15 06:27:14,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17819.50 MB 2025-02-15 06:27:14,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27019.54 MB 2025-02-15 06:27:14,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:27:14,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:27:14,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:27:14,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:14,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23040.99 MB 2025-02-15 06:27:14,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24930.53 MB 2025-02-15 06:27:14,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:27:14,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29058.14 MB 2025-02-15 06:27:14,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29058.14 MB 2025-02-15 06:27:14,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:27:14,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26347.96 MB 2025-02-15 06:27:15,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:27:15,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:27:15,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:27:15,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24930.53 MB 2025-02-15 06:27:15,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27172.38 MB 2025-02-15 06:27:15,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:27:15,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29058.14 MB 2025-02-15 06:27:15,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 06:27:15,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:27:15,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32716.66 MB 2025-02-15 06:27:15,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:27:15,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:27:15,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:27:15,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23040.99 MB 2025-02-15 06:27:15,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27172.38 MB 2025-02-15 06:27:15,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:27:15,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29058.14 MB 2025-02-15 06:27:15,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 06:27:15,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:27:15,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32716.66 MB 2025-02-15 06:27:15,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:27:15,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:27:15,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:27:15,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28705.93 MB 2025-02-15 06:27:15,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29472.93 MB 2025-02-15 06:27:15,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:27:15,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34720.45 MB 2025-02-15 06:27:15,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 06:27:15,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 06:27:15,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30180.72 MB 2025-02-15 06:27:15,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:27:15,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:27:15,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:27:15,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29885.82 MB 2025-02-15 06:27:15,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30113.94 MB 2025-02-15 06:27:15,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-15 06:27:15,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 06:27:15,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 06:27:15,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:27:15,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30344.94 MB 2025-02-15 06:27:15,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:27:15,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:27:15,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.36 seconds 2025-02-15 06:27:15,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17480.59 MB 2025-02-15 06:27:15,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30314.79 MB 2025-02-15 06:27:15,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12834.20 MB 2025-02-15 06:27:15,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54775.51 MB 2025-02-15 06:27:15,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 06:27:15,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19639.83 MB 2025-02-15 06:27:15,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30344.94 MB 2025-02-15 06:27:15,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:27:15,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:27:15,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:27:15,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30314.79 MB 2025-02-15 06:27:15,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22469.79 MB 2025-02-15 06:27:15,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7845.00 MB 2025-02-15 06:27:15,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 06:27:15,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35135.68 MB 2025-02-15 06:27:15,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:27:15,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32813.56 MB 2025-02-15 06:27:15,509 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 06:27:15,509 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:27:15,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:27:15,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:27:15,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:27:15,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:27:15,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22469.79 MB 2025-02-15 06:27:15,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30865.52 MB 2025-02-15 06:27:15,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.73 MB 2025-02-15 06:27:15,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35135.68 MB 2025-02-15 06:27:15,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-15 06:27:15,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 06:27:15,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30865.52 MB 2025-02-15 06:27:15,672 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 06:27:15,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:27:15,673 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:27:15,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:27:15,674 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:27:15,679 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:27:15,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:27:15,680 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:27:15,680 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:28:07,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:07,729 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:28:07,734 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:28:07,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:07,739 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 876, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:28:07,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:07,740 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 876, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:28:21,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:28:21,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:28:21,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.55 seconds 2025-02-15 06:28:21,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:21,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19072.82 MB 2025-02-15 06:28:21,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22172.93 MB 2025-02-15 06:28:21,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3100.11 MB 2025-02-15 06:28:21,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 06:28:21,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27950.84 MB 2025-02-15 06:28:21,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19704.84 MB 2025-02-15 06:28:21,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31035.60 MB 2025-02-15 06:28:21,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:28:21,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:28:21,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 06:28:21,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:21,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22172.93 MB 2025-02-15 06:28:21,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20332.95 MB 2025-02-15 06:28:21,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1839.98 MB 2025-02-15 06:28:21,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27950.84 MB 2025-02-15 06:28:21,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-15 06:28:21,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 06:28:21,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31821.06 MB 2025-02-15 06:28:23,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:28:23,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:28:23,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:28:23,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20332.95 MB 2025-02-15 06:28:23,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20863.79 MB 2025-02-15 06:28:23,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:28:23,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35500.59 MB 2025-02-15 06:28:23,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26973.57 MB 2025-02-15 06:28:23,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8527.02 MB 2025-02-15 06:28:23,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24842.34 MB 2025-02-15 06:28:23,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:28:23,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:28:23,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:28:23,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20863.79 MB 2025-02-15 06:28:23,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22753.33 MB 2025-02-15 06:28:23,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:28:23,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26973.57 MB 2025-02-15 06:28:23,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26973.57 MB 2025-02-15 06:28:23,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:23,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24170.76 MB 2025-02-15 06:28:23,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:28:23,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:28:23,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:28:23,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22753.33 MB 2025-02-15 06:28:23,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24995.18 MB 2025-02-15 06:28:23,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:28:23,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26973.57 MB 2025-02-15 06:28:23,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32635.88 MB 2025-02-15 06:28:23,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:28:23,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30539.46 MB 2025-02-15 06:28:23,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:28:23,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:28:23,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:28:23,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20863.79 MB 2025-02-15 06:28:23,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24995.18 MB 2025-02-15 06:28:23,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:28:23,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26973.57 MB 2025-02-15 06:28:23,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32635.88 MB 2025-02-15 06:28:23,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:28:23,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30539.46 MB 2025-02-15 06:28:23,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:28:23,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:28:23,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:28:23,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26528.72 MB 2025-02-15 06:28:23,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27295.73 MB 2025-02-15 06:28:23,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:28:23,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32635.88 MB 2025-02-15 06:28:23,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-15 06:28:23,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:28:23,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28003.51 MB 2025-02-15 06:28:23,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:28:23,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:28:23,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:28:23,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27708.62 MB 2025-02-15 06:28:23,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27937.32 MB 2025-02-15 06:28:23,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.70 MB 2025-02-15 06:28:23,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33049.02 MB 2025-02-15 06:28:23,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-15 06:28:23,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:23,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28137.55 MB 2025-02-15 06:28:23,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:28:23,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:28:23,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.98 seconds 2025-02-15 06:28:23,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16020.76 MB 2025-02-15 06:28:23,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28138.17 MB 2025-02-15 06:28:23,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12117.41 MB 2025-02-15 06:28:23,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 06:28:23,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-15 06:28:23,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14606.66 MB 2025-02-15 06:28:23,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28138.17 MB 2025-02-15 06:28:23,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:28:23,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:28:23,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:28:23,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:23,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28138.17 MB 2025-02-15 06:28:23,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21008.18 MB 2025-02-15 06:28:23,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7129.99 MB 2025-02-15 06:28:23,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33049.02 MB 2025-02-15 06:28:23,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-15 06:28:23,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:23,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30635.40 MB 2025-02-15 06:28:24,008 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 06:28:24,008 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:28:24,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:28:24,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:28:24,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:28:24,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:24,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21008.18 MB 2025-02-15 06:28:24,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29399.22 MB 2025-02-15 06:28:24,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 06:28:24,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33049.02 MB 2025-02-15 06:28:24,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41391.49 MB 2025-02-15 06:28:24,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 06:28:24,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29399.22 MB 2025-02-15 06:28:24,174 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 06:28:24,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:24,176 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:28:24,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:24,177 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:28:24,181 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:28:24,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:24,183 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:28:24,183 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:28:33,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:33,792 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:28:33,797 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:28:33,802 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:33,802 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1376, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:28:33,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:33,803 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1376, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:28:55,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:28:55,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:28:55,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.47 seconds 2025-02-15 06:28:55,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:55,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22556.89 MB 2025-02-15 06:28:55,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27426.48 MB 2025-02-15 06:28:55,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4869.59 MB 2025-02-15 06:28:55,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49733.96 MB 2025-02-15 06:28:55,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38036.05 MB 2025-02-15 06:28:55,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11697.91 MB 2025-02-15 06:28:55,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36331.62 MB 2025-02-15 06:28:55,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:28:55,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:28:55,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:28:55,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:55,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.48 MB 2025-02-15 06:28:55,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22931.25 MB 2025-02-15 06:28:55,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4495.24 MB 2025-02-15 06:28:55,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38036.05 MB 2025-02-15 06:28:55,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47588.57 MB 2025-02-15 06:28:55,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9552.53 MB 2025-02-15 06:28:55,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41792.01 MB 2025-02-15 06:28:57,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:28:57,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:28:57,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:28:57,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22931.25 MB 2025-02-15 06:28:57,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23462.09 MB 2025-02-15 06:28:57,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:28:57,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47588.57 MB 2025-02-15 06:28:57,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33166.46 MB 2025-02-15 06:28:57,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14422.11 MB 2025-02-15 06:28:57,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27440.63 MB 2025-02-15 06:28:57,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:28:57,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:28:57,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:28:57,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-15 06:28:57,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25351.62 MB 2025-02-15 06:28:57,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:28:57,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33166.46 MB 2025-02-15 06:28:57,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33166.46 MB 2025-02-15 06:28:57,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:57,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26769.05 MB 2025-02-15 06:28:57,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:28:57,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:28:57,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:28:57,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25351.62 MB 2025-02-15 06:28:57,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-15 06:28:57,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:28:57,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33166.46 MB 2025-02-15 06:28:57,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35997.61 MB 2025-02-15 06:28:57,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:28:57,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-15 06:28:57,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:28:57,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:28:57,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 06:28:57,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-15 06:28:57,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-15 06:28:57,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:28:57,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33166.46 MB 2025-02-15 06:28:57,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35997.61 MB 2025-02-15 06:28:57,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:28:57,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-15 06:28:57,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:28:57,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:28:57,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:28:57,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29127.02 MB 2025-02-15 06:28:57,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29894.02 MB 2025-02-15 06:28:57,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:28:57,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35997.61 MB 2025-02-15 06:28:57,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-15 06:28:57,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 06:28:57,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30601.81 MB 2025-02-15 06:28:57,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:28:57,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:28:57,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:28:57,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30306.91 MB 2025-02-15 06:28:57,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30534.32 MB 2025-02-15 06:28:57,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.41 MB 2025-02-15 06:28:57,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36412.85 MB 2025-02-15 06:28:57,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-15 06:28:57,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:57,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30762.55 MB 2025-02-15 06:28:57,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:28:57,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:28:57,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.91 seconds 2025-02-15 06:28:57,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.80 MB 2025-02-15 06:28:57,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30735.17 MB 2025-02-15 06:28:57,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12972.37 MB 2025-02-15 06:28:57,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49733.96 MB 2025-02-15 06:28:57,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-15 06:28:57,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13321.11 MB 2025-02-15 06:28:57,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30762.55 MB 2025-02-15 06:28:57,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:28:57,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:28:57,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:28:57,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:57,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30735.17 MB 2025-02-15 06:28:57,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22759.84 MB 2025-02-15 06:28:57,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7975.33 MB 2025-02-15 06:28:57,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36412.85 MB 2025-02-15 06:28:57,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-15 06:28:57,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:28:57,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33240.70 MB 2025-02-15 06:28:58,006 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 06:28:58,007 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:28:58,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:28:58,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:28:58,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:28:58,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:28:58,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22759.84 MB 2025-02-15 06:28:58,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31178.00 MB 2025-02-15 06:28:58,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 06:28:58,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36412.85 MB 2025-02-15 06:28:58,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44782.58 MB 2025-02-15 06:28:58,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-15 06:28:58,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31178.00 MB 2025-02-15 06:28:58,171 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 06:28:58,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:58,173 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:28:58,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:58,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:28:58,178 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:28:58,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:28:58,179 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:28:58,179 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:29:57,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:29:57,059 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:29:57,064 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:29:57,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:29:57,068 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:29:57,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:29:57,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:29:59,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:29:59,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:29:59,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-15 06:29:59,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:29:59,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14041.80 MB 2025-02-15 06:29:59,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14586.80 MB 2025-02-15 06:29:59,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-15 06:29:59,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57336.14 MB 2025-02-15 06:29:59,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 06:29:59,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38967.18 MB 2025-02-15 06:29:59,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23513.17 MB 2025-02-15 06:29:59,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:29:59,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:29:59,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:29:59,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:29:59,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14586.80 MB 2025-02-15 06:29:59,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14710.39 MB 2025-02-15 06:29:59,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.59 MB 2025-02-15 06:29:59,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 06:29:59,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 06:29:59,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:29:59,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16469.04 MB 2025-02-15 06:30:00,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:30:00,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:30:00,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.65 seconds 2025-02-15 06:30:00,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14710.39 MB 2025-02-15 06:30:00,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14888.22 MB 2025-02-15 06:30:00,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-15 06:30:00,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 06:30:00,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17897.10 MB 2025-02-15 06:30:00,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 06:30:00,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18881.08 MB 2025-02-15 06:30:00,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:30:00,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:30:00,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:30:00,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-15 06:30:00,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15521.00 MB 2025-02-15 06:30:00,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-15 06:30:00,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 06:30:00,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17897.10 MB 2025-02-15 06:30:00,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:30:00,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15995.84 MB 2025-02-15 06:30:00,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:30:00,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:30:00,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:30:00,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15521.00 MB 2025-02-15 06:30:00,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-15 06:30:00,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-15 06:30:00,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 06:30:00,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19163.77 MB 2025-02-15 06:30:00,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1266.68 MB 2025-02-15 06:30:00,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18131.45 MB 2025-02-15 06:30:00,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:30:00,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:30:00,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:30:00,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-15 06:30:00,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-15 06:30:00,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-15 06:30:00,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 06:30:00,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19163.77 MB 2025-02-15 06:30:00,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1266.68 MB 2025-02-15 06:30:00,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18131.45 MB 2025-02-15 06:30:00,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:30:00,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:30:00,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 06:30:00,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16785.80 MB 2025-02-15 06:30:00,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.74 MB 2025-02-15 06:30:00,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-15 06:30:00,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19163.77 MB 2025-02-15 06:30:00,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-15 06:30:00,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-15 06:30:00,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17292.10 MB 2025-02-15 06:30:00,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:30:00,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:30:00,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:30:00,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17181.07 MB 2025-02-15 06:30:00,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17400.63 MB 2025-02-15 06:30:00,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.56 MB 2025-02-15 06:30:00,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19297.99 MB 2025-02-15 06:30:00,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-15 06:30:00,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:30:00,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17400.63 MB 2025-02-15 06:30:00,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:30:00,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:30:00,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-15 06:30:00,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13505.25 MB 2025-02-15 06:30:00,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14239.77 MB 2025-02-15 06:30:00,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.51 MB 2025-02-15 06:30:00,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57336.14 MB 2025-02-15 06:30:00,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-15 06:30:00,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38038.14 MB 2025-02-15 06:30:00,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17601.34 MB 2025-02-15 06:30:00,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:30:00,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:30:00,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:30:00,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14239.77 MB 2025-02-15 06:30:00,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17248.42 MB 2025-02-15 06:30:00,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.65 MB 2025-02-15 06:30:00,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19297.99 MB 2025-02-15 06:30:00,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-15 06:30:00,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:30:00,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17549.23 MB 2025-02-15 06:30:00,599 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 06:30:00,599 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:30:00,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:30:00,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:30:00,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:30:00,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:30:00,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17248.42 MB 2025-02-15 06:30:00,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.62 MB 2025-02-15 06:30:00,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 06:30:00,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19297.99 MB 2025-02-15 06:30:00,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29769.07 MB 2025-02-15 06:30:00,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 06:30:00,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25671.62 MB 2025-02-15 06:30:00,770 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 06:30:00,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:30:00,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:30:00,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:30:00,772 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:30:00,777 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:30:00,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:30:00,778 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:30:00,779 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:31:33,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:33,061 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:31:33,067 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:31:33,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:33,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:31:33,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:33,072 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:31:52,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:31:52,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:31:52,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.32 seconds 2025-02-15 06:31:52,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:52,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-15 06:31:52,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-15 06:31:52,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-15 06:31:52,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38145.10 MB 2025-02-15 06:31:52,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37708.89 MB 2025-02-15 06:31:52,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -436.21 MB 2025-02-15 06:31:52,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-15 06:31:52,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:31:52,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:31:52,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:31:52,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:52,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-15 06:31:52,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-15 06:31:52,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-15 06:31:52,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37708.89 MB 2025-02-15 06:31:52,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46596.62 MB 2025-02-15 06:31:52,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8887.73 MB 2025-02-15 06:31:52,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39623.18 MB 2025-02-15 06:31:54,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:31:54,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:31:54,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:31:54,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-15 06:31:54,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-15 06:31:54,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:31:54,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46596.62 MB 2025-02-15 06:31:54,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29043.46 MB 2025-02-15 06:31:54,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17553.16 MB 2025-02-15 06:31:54,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.58 MB 2025-02-15 06:31:54,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:31:54,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:31:54,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:31:54,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-15 06:31:54,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-15 06:31:54,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:31:54,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29043.46 MB 2025-02-15 06:31:54,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29043.46 MB 2025-02-15 06:31:54,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:31:54,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-15 06:31:54,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:31:54,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:31:54,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:31:54,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-15 06:31:54,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-15 06:31:54,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:31:54,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29043.46 MB 2025-02-15 06:31:54,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34705.77 MB 2025-02-15 06:31:54,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:31:54,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-15 06:31:54,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:31:54,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:31:54,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:31:54,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-15 06:31:54,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-15 06:31:54,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:31:54,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29043.46 MB 2025-02-15 06:31:54,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34705.77 MB 2025-02-15 06:31:54,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:31:54,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-15 06:31:54,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:31:54,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:31:54,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:31:54,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-15 06:31:54,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-15 06:31:54,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:31:54,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34705.77 MB 2025-02-15 06:31:54,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 06:31:54,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 06:31:54,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-15 06:31:54,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:31:54,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:31:54,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:31:54,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-15 06:31:54,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.79 MB 2025-02-15 06:31:54,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-15 06:31:54,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 06:31:54,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 06:31:54,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:31:54,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30186.09 MB 2025-02-15 06:31:54,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:31:54,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:31:54,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.76 seconds 2025-02-15 06:31:54,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:54,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-15 06:31:54,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30159.64 MB 2025-02-15 06:31:54,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.58 MB 2025-02-15 06:31:54,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38145.10 MB 2025-02-15 06:31:54,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 06:31:54,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3024.09 MB 2025-02-15 06:31:54,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30186.09 MB 2025-02-15 06:31:55,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:31:55,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:31:55,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:31:55,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:55,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30159.64 MB 2025-02-15 06:31:55,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.91 MB 2025-02-15 06:31:55,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7794.73 MB 2025-02-15 06:31:55,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 06:31:55,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 06:31:55,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:31:55,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32658.10 MB 2025-02-15 06:31:55,118 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 06:31:55,118 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:31:55,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:31:55,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:31:55,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:31:55,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:31:55,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.91 MB 2025-02-15 06:31:55,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30760.13 MB 2025-02-15 06:31:55,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 06:31:55,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 06:31:55,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39294.34 MB 2025-02-15 06:31:55,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 06:31:55,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30760.13 MB 2025-02-15 06:31:55,286 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 06:31:55,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:55,288 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:31:55,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:55,289 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:31:55,293 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:31:55,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:31:55,295 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:31:55,295 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:33:01,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:01,100 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:33:01,108 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:33:01,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:01,115 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:33:01,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:01,117 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:33:33,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:33:33,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:33:33,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.62 seconds 2025-02-15 06:33:33,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:33,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27664.56 MB 2025-02-15 06:33:33,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35128.32 MB 2025-02-15 06:33:33,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7463.76 MB 2025-02-15 06:33:33,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47641.00 MB 2025-02-15 06:33:33,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40649.10 MB 2025-02-15 06:33:33,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6991.90 MB 2025-02-15 06:33:33,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43930.70 MB 2025-02-15 06:33:33,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:33:33,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:33:33,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:33:33,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:33,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35128.32 MB 2025-02-15 06:33:33,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.93 MB 2025-02-15 06:33:33,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8385.39 MB 2025-02-15 06:33:33,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40649.10 MB 2025-02-15 06:33:33,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55650.03 MB 2025-02-15 06:33:33,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15000.93 MB 2025-02-15 06:33:33,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54496.76 MB 2025-02-15 06:33:35,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:33:35,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:33:35,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:33:35,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:35,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.93 MB 2025-02-15 06:33:35,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27273.77 MB 2025-02-15 06:33:35,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:33:35,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55650.03 MB 2025-02-15 06:33:35,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31136.42 MB 2025-02-15 06:33:35,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24513.61 MB 2025-02-15 06:33:35,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31252.47 MB 2025-02-15 06:33:35,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:33:35,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:33:35,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:33:35,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:35,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27273.77 MB 2025-02-15 06:33:35,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29163.31 MB 2025-02-15 06:33:35,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:33:35,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31136.42 MB 2025-02-15 06:33:35,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33023.85 MB 2025-02-15 06:33:35,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:33:35,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30580.74 MB 2025-02-15 06:33:36,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:33:36,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:33:36,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-15 06:33:36,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29163.31 MB 2025-02-15 06:33:36,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31405.16 MB 2025-02-15 06:33:36,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:33:36,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33023.85 MB 2025-02-15 06:33:36,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38686.16 MB 2025-02-15 06:33:36,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:33:36,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36949.45 MB 2025-02-15 06:33:36,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:33:36,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:33:36,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.36 seconds 2025-02-15 06:33:36,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27273.77 MB 2025-02-15 06:33:36,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31405.16 MB 2025-02-15 06:33:36,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:33:36,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31136.42 MB 2025-02-15 06:33:36,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38686.16 MB 2025-02-15 06:33:36,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 06:33:36,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36949.45 MB 2025-02-15 06:33:36,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:33:36,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:33:36,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:33:36,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32938.71 MB 2025-02-15 06:33:36,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33705.71 MB 2025-02-15 06:33:36,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:33:36,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38686.16 MB 2025-02-15 06:33:36,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39103.50 MB 2025-02-15 06:33:36,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 06:33:36,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34413.50 MB 2025-02-15 06:33:36,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:33:36,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:33:36,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:33:36,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34118.60 MB 2025-02-15 06:33:36,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34346.97 MB 2025-02-15 06:33:36,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 06:33:36,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39103.50 MB 2025-02-15 06:33:36,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39103.50 MB 2025-02-15 06:33:36,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:33:36,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34570.51 MB 2025-02-15 06:33:36,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:33:36,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:33:36,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.31 seconds 2025-02-15 06:33:36,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20316.63 MB 2025-02-15 06:33:36,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34547.25 MB 2025-02-15 06:33:36,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14230.62 MB 2025-02-15 06:33:36,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47641.00 MB 2025-02-15 06:33:36,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39103.50 MB 2025-02-15 06:33:36,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8537.51 MB 2025-02-15 06:33:36,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34570.51 MB 2025-02-15 06:33:36,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:33:36,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:33:36,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:33:36,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34547.25 MB 2025-02-15 06:33:36,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25309.40 MB 2025-02-15 06:33:36,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9237.86 MB 2025-02-15 06:33:36,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39103.50 MB 2025-02-15 06:33:36,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39103.50 MB 2025-02-15 06:33:36,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:33:36,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34547.25 MB 2025-02-15 06:33:36,716 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 06:33:36,717 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:33:36,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:33:36,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:33:36,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:33:36,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:33:36,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25309.40 MB 2025-02-15 06:33:36,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33715.06 MB 2025-02-15 06:33:36,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 06:33:36,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39103.50 MB 2025-02-15 06:33:36,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47462.74 MB 2025-02-15 06:33:36,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 06:33:36,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33715.06 MB 2025-02-15 06:33:36,880 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 06:33:36,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:36,882 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:33:36,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:36,883 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:33:36,888 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:33:36,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:33:36,889 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:33:36,889 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:34:26,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:26,520 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:34:26,525 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:34:26,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:26,528 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1561, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:34:26,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:26,529 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1561, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:34:50,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:34:50,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:34:50,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.20 seconds 2025-02-15 06:34:50,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:50,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36760.21 MB 2025-02-15 06:34:50,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42284.50 MB 2025-02-15 06:34:50,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5524.29 MB 2025-02-15 06:34:50,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55821.99 MB 2025-02-15 06:34:50,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53619.98 MB 2025-02-15 06:34:50,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2202.01 MB 2025-02-15 06:34:50,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51214.41 MB 2025-02-15 06:34:50,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:34:50,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:34:50,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:34:50,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:50,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42284.50 MB 2025-02-15 06:34:50,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36807.21 MB 2025-02-15 06:34:50,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5477.29 MB 2025-02-15 06:34:50,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53619.98 MB 2025-02-15 06:34:50,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64107.84 MB 2025-02-15 06:34:50,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10487.86 MB 2025-02-15 06:34:50,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58054.87 MB 2025-02-15 06:34:52,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:34:52,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:34:52,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:34:52,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:52,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36807.21 MB 2025-02-15 06:34:52,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37338.05 MB 2025-02-15 06:34:52,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:34:52,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64107.84 MB 2025-02-15 06:34:52,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43914.36 MB 2025-02-15 06:34:52,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20193.48 MB 2025-02-15 06:34:52,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41316.59 MB 2025-02-15 06:34:52,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:34:52,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:34:52,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:34:52,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:52,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37338.05 MB 2025-02-15 06:34:52,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39227.58 MB 2025-02-15 06:34:52,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:34:52,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43914.36 MB 2025-02-15 06:34:52,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44858.08 MB 2025-02-15 06:34:52,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:34:52,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40645.01 MB 2025-02-15 06:34:52,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:34:52,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:34:52,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:34:52,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:52,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39227.58 MB 2025-02-15 06:34:52,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41469.44 MB 2025-02-15 06:34:52,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:34:52,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44858.08 MB 2025-02-15 06:34:52,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50520.39 MB 2025-02-15 06:34:52,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:34:52,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47013.72 MB 2025-02-15 06:34:52,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:34:52,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:34:52,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:34:52,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:52,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37338.05 MB 2025-02-15 06:34:52,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41469.44 MB 2025-02-15 06:34:52,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:34:52,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43914.36 MB 2025-02-15 06:34:52,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50520.39 MB 2025-02-15 06:34:52,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:34:52,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47013.72 MB 2025-02-15 06:34:53,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:34:53,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:34:53,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:34:53,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:53,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43002.98 MB 2025-02-15 06:34:53,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43769.98 MB 2025-02-15 06:34:53,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:34:53,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50520.39 MB 2025-02-15 06:34:53,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50933.53 MB 2025-02-15 06:34:53,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:34:53,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44477.77 MB 2025-02-15 06:34:53,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:34:53,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:34:53,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:34:53,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:53,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44182.87 MB 2025-02-15 06:34:53,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44410.72 MB 2025-02-15 06:34:53,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.85 MB 2025-02-15 06:34:53,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50933.53 MB 2025-02-15 06:34:53,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50933.53 MB 2025-02-15 06:34:53,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:34:53,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44650.89 MB 2025-02-15 06:34:53,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:34:53,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:34:53,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.65 seconds 2025-02-15 06:34:53,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:53,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31321.56 MB 2025-02-15 06:34:53,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44611.58 MB 2025-02-15 06:34:53,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13290.02 MB 2025-02-15 06:34:53,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55821.99 MB 2025-02-15 06:34:53,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50933.53 MB 2025-02-15 06:34:53,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4888.46 MB 2025-02-15 06:34:53,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44650.89 MB 2025-02-15 06:34:53,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:34:53,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:34:53,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:34:53,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:53,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44611.58 MB 2025-02-15 06:34:53,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36314.14 MB 2025-02-15 06:34:53,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8297.44 MB 2025-02-15 06:34:53,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50933.53 MB 2025-02-15 06:34:53,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50933.53 MB 2025-02-15 06:34:53,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:34:53,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44611.58 MB 2025-02-15 06:34:53,475 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 06:34:53,475 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:34:53,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:34:53,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:34:53,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:34:53,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:34:53,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36314.14 MB 2025-02-15 06:34:53,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44721.87 MB 2025-02-15 06:34:53,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 06:34:53,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50933.53 MB 2025-02-15 06:34:53,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55113.15 MB 2025-02-15 06:34:53,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 06:34:53,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44721.87 MB 2025-02-15 06:34:53,643 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 06:34:53,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:53,645 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:34:53,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:53,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:34:53,650 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:34:53,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:34:53,651 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:34:53,652 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:35:51,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:35:51,453 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:35:51,459 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:35:51,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:35:51,463 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:35:51,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:35:51,464 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:36:10,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:36:10,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:36:10,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.13 seconds 2025-02-15 06:36:10,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:10,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34516.46 MB 2025-02-15 06:36:10,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38901.60 MB 2025-02-15 06:36:10,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4385.14 MB 2025-02-15 06:36:10,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63472.40 MB 2025-02-15 06:36:10,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52481.23 MB 2025-02-15 06:36:10,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10991.17 MB 2025-02-15 06:36:10,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47838.20 MB 2025-02-15 06:36:10,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:36:10,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:36:10,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 06:36:10,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:10,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38901.60 MB 2025-02-15 06:36:10,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34768.03 MB 2025-02-15 06:36:10,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4133.57 MB 2025-02-15 06:36:10,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52481.23 MB 2025-02-15 06:36:10,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52481.23 MB 2025-02-15 06:36:10,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:36:10,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43789.63 MB 2025-02-15 06:36:12,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:36:12,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:36:12,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.67 seconds 2025-02-15 06:36:12,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34768.03 MB 2025-02-15 06:36:12,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35229.86 MB 2025-02-15 06:36:12,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 461.83 MB 2025-02-15 06:36:12,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52481.23 MB 2025-02-15 06:36:12,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 06:36:12,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4385.14 MB 2025-02-15 06:36:12,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39192.48 MB 2025-02-15 06:36:12,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:36:12,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:36:12,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:36:12,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35229.86 MB 2025-02-15 06:36:12,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36873.49 MB 2025-02-15 06:36:12,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1643.63 MB 2025-02-15 06:36:12,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 06:36:12,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 06:36:12,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:36:12,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38106.65 MB 2025-02-15 06:36:12,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:36:12,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:36:12,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 06:36:12,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36873.49 MB 2025-02-15 06:36:12,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38823.91 MB 2025-02-15 06:36:12,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1950.42 MB 2025-02-15 06:36:12,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 06:36:12,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 06:36:12,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:36:12,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43647.43 MB 2025-02-15 06:36:12,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:36:12,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:36:12,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:36:12,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35229.86 MB 2025-02-15 06:36:12,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38823.91 MB 2025-02-15 06:36:12,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3594.05 MB 2025-02-15 06:36:12,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 06:36:12,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 06:36:12,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:36:12,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43647.43 MB 2025-02-15 06:36:12,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:36:12,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:36:12,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 06:36:12,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40158.09 MB 2025-02-15 06:36:12,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40825.39 MB 2025-02-15 06:36:12,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 667.29 MB 2025-02-15 06:36:12,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 06:36:12,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48452.60 MB 2025-02-15 06:36:12,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 356.52 MB 2025-02-15 06:36:12,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41441.16 MB 2025-02-15 06:36:12,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:36:12,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:36:12,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:36:12,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41184.60 MB 2025-02-15 06:36:12,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41402.53 MB 2025-02-15 06:36:12,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.92 MB 2025-02-15 06:36:12,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48452.60 MB 2025-02-15 06:36:12,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 06:36:12,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 06:36:12,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41524.76 MB 2025-02-15 06:36:12,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:36:12,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:36:12,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.24 seconds 2025-02-15 06:36:12,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30199.68 MB 2025-02-15 06:36:12,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41603.60 MB 2025-02-15 06:36:12,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11403.91 MB 2025-02-15 06:36:12,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63472.40 MB 2025-02-15 06:36:12,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 06:36:12,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15017.71 MB 2025-02-15 06:36:12,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41603.60 MB 2025-02-15 06:36:12,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:36:12,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:36:12,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:36:12,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:12,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41603.60 MB 2025-02-15 06:36:12,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44617.63 MB 2025-02-15 06:36:12,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 06:36:12,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48454.70 MB 2025-02-15 06:36:12,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 06:36:12,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:36:12,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44919.00 MB 2025-02-15 06:36:12,993 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:36:12,993 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:36:13,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:36:13,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:36:13,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:36:13,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:36:13,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34958.67 MB 2025-02-15 06:36:13,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43397.69 MB 2025-02-15 06:36:13,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:36:13,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48454.70 MB 2025-02-15 06:36:13,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56845.40 MB 2025-02-15 06:36:13,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:36:13,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43397.69 MB 2025-02-15 06:36:13,162 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:36:13,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:36:13,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:36:13,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:36:13,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:36:13,169 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:36:13,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:36:13,170 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:36:13,170 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:37:08,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:08,787 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:37:08,792 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:37:08,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:08,797 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:37:08,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:08,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:37:27,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:37:27,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:37:27,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.53 seconds 2025-02-15 06:37:27,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:27,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34223.80 MB 2025-02-15 06:37:27,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38460.04 MB 2025-02-15 06:37:27,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4236.25 MB 2025-02-15 06:37:27,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69430.41 MB 2025-02-15 06:37:27,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43973.08 MB 2025-02-15 06:37:27,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25457.33 MB 2025-02-15 06:37:27,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47319.85 MB 2025-02-15 06:37:27,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:37:27,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:37:27,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:37:27,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:27,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38460.04 MB 2025-02-15 06:37:27,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34914.88 MB 2025-02-15 06:37:27,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3545.16 MB 2025-02-15 06:37:27,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43973.08 MB 2025-02-15 06:37:27,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53355.74 MB 2025-02-15 06:37:27,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9382.66 MB 2025-02-15 06:37:27,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50650.12 MB 2025-02-15 06:37:29,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:37:29,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:37:29,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:37:29,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34914.88 MB 2025-02-15 06:37:29,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35445.72 MB 2025-02-15 06:37:29,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:37:29,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53355.74 MB 2025-02-15 06:37:29,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41152.41 MB 2025-02-15 06:37:29,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12203.33 MB 2025-02-15 06:37:29,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39424.27 MB 2025-02-15 06:37:29,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:37:29,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:37:29,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:37:29,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35445.72 MB 2025-02-15 06:37:29,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37335.26 MB 2025-02-15 06:37:29,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:37:29,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41152.41 MB 2025-02-15 06:37:29,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42096.13 MB 2025-02-15 06:37:29,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:37:29,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38752.69 MB 2025-02-15 06:37:29,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:37:29,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:37:29,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:37:29,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37335.26 MB 2025-02-15 06:37:29,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39578.15 MB 2025-02-15 06:37:29,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.89 MB 2025-02-15 06:37:29,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42096.13 MB 2025-02-15 06:37:29,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47758.44 MB 2025-02-15 06:37:29,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:37:29,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45122.43 MB 2025-02-15 06:37:29,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:37:29,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:37:29,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:37:29,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35445.72 MB 2025-02-15 06:37:29,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39578.15 MB 2025-02-15 06:37:29,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.42 MB 2025-02-15 06:37:29,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41152.41 MB 2025-02-15 06:37:29,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47758.44 MB 2025-02-15 06:37:29,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:37:29,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45122.43 MB 2025-02-15 06:37:29,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:37:29,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:37:29,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:37:29,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41111.69 MB 2025-02-15 06:37:29,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41878.69 MB 2025-02-15 06:37:29,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:37:29,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47758.44 MB 2025-02-15 06:37:29,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48169.48 MB 2025-02-15 06:37:29,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 06:37:29,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42586.48 MB 2025-02-15 06:37:29,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:37:29,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:37:29,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:37:29,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42291.58 MB 2025-02-15 06:37:29,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42520.02 MB 2025-02-15 06:37:29,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 06:37:29,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48169.48 MB 2025-02-15 06:37:29,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48169.48 MB 2025-02-15 06:37:29,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:37:29,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42735.54 MB 2025-02-15 06:37:29,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:37:29,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:37:29,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.97 seconds 2025-02-15 06:37:29,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:29,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30053.35 MB 2025-02-15 06:37:29,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42720.93 MB 2025-02-15 06:37:29,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12667.57 MB 2025-02-15 06:37:29,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69430.41 MB 2025-02-15 06:37:29,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48169.48 MB 2025-02-15 06:37:29,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21260.93 MB 2025-02-15 06:37:29,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42735.54 MB 2025-02-15 06:37:30,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:37:30,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:37:30,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:37:30,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:30,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42720.93 MB 2025-02-15 06:37:30,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35054.53 MB 2025-02-15 06:37:30,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7666.39 MB 2025-02-15 06:37:30,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48169.48 MB 2025-02-15 06:37:30,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48169.48 MB 2025-02-15 06:37:30,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:37:30,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45230.44 MB 2025-02-15 06:37:30,055 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 06:37:30,055 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:37:30,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:37:30,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:37:30,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:37:30,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:37:30,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35054.53 MB 2025-02-15 06:37:30,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43486.00 MB 2025-02-15 06:37:30,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 06:37:30,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48169.48 MB 2025-02-15 06:37:30,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56553.90 MB 2025-02-15 06:37:30,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 06:37:30,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43486.00 MB 2025-02-15 06:37:30,221 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 06:37:30,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:30,223 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:37:30,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:30,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:37:30,228 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:37:30,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:37:30,229 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:37:30,229 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:38:17,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:17,845 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:38:17,850 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:38:17,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:17,854 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:38:17,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:17,855 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:38:37,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:38:37,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:38:37,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.58 seconds 2025-02-15 06:38:37,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:37,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34697.63 MB 2025-02-15 06:38:37,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39175.05 MB 2025-02-15 06:38:37,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-15 06:38:37,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 06:38:37,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52600.77 MB 2025-02-15 06:38:37,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12337.55 MB 2025-02-15 06:38:37,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48019.37 MB 2025-02-15 06:38:37,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:38:37,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:38:37,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:38:37,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:37,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39175.05 MB 2025-02-15 06:38:37,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35268.39 MB 2025-02-15 06:38:37,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-15 06:38:37,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52600.77 MB 2025-02-15 06:38:37,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61320.72 MB 2025-02-15 06:38:37,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8719.96 MB 2025-02-15 06:38:37,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52284.02 MB 2025-02-15 06:38:39,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:38:39,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:38:39,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:38:39,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35268.39 MB 2025-02-15 06:38:39,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35799.24 MB 2025-02-15 06:38:39,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:38:39,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61320.72 MB 2025-02-15 06:38:39,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48123.35 MB 2025-02-15 06:38:39,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13197.38 MB 2025-02-15 06:38:39,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39777.78 MB 2025-02-15 06:38:39,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:38:39,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:38:39,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:38:39,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35799.24 MB 2025-02-15 06:38:39,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37688.77 MB 2025-02-15 06:38:39,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:38:39,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48123.35 MB 2025-02-15 06:38:39,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48123.35 MB 2025-02-15 06:38:39,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:38:39,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39106.20 MB 2025-02-15 06:38:39,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:38:39,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:38:39,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:38:39,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37688.77 MB 2025-02-15 06:38:39,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39930.63 MB 2025-02-15 06:38:39,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:38:39,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48123.35 MB 2025-02-15 06:38:39,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49067.07 MB 2025-02-15 06:38:39,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:38:39,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45474.91 MB 2025-02-15 06:38:39,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:38:39,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:38:39,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:38:39,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35799.24 MB 2025-02-15 06:38:39,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39930.63 MB 2025-02-15 06:38:39,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:38:39,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48123.35 MB 2025-02-15 06:38:39,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49067.07 MB 2025-02-15 06:38:39,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:38:39,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45474.91 MB 2025-02-15 06:38:39,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:38:39,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:38:39,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:38:39,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41464.17 MB 2025-02-15 06:38:39,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42231.17 MB 2025-02-15 06:38:39,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:38:39,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49067.07 MB 2025-02-15 06:38:39,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49476.01 MB 2025-02-15 06:38:39,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 06:38:39,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42938.96 MB 2025-02-15 06:38:39,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:38:39,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:38:39,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:38:39,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42644.06 MB 2025-02-15 06:38:39,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42872.44 MB 2025-02-15 06:38:39,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.38 MB 2025-02-15 06:38:39,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49476.01 MB 2025-02-15 06:38:39,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49476.01 MB 2025-02-15 06:38:39,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:38:39,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43083.42 MB 2025-02-15 06:38:39,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:38:39,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:38:39,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.00 seconds 2025-02-15 06:38:39,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:39,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30290.27 MB 2025-02-15 06:38:39,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43073.19 MB 2025-02-15 06:38:39,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12782.92 MB 2025-02-15 06:38:39,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 06:38:39,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49476.01 MB 2025-02-15 06:38:39,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15462.30 MB 2025-02-15 06:38:39,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43083.42 MB 2025-02-15 06:38:40,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:38:40,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:38:40,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:38:40,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:40,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43073.19 MB 2025-02-15 06:38:40,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35289.17 MB 2025-02-15 06:38:40,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7784.03 MB 2025-02-15 06:38:40,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49476.01 MB 2025-02-15 06:38:40,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49476.01 MB 2025-02-15 06:38:40,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:38:40,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45580.87 MB 2025-02-15 06:38:40,150 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 06:38:40,151 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:38:40,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:38:40,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:38:40,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 06:38:40,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:38:40,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35289.17 MB 2025-02-15 06:38:40,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43715.35 MB 2025-02-15 06:38:40,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 06:38:40,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49476.01 MB 2025-02-15 06:38:40,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57852.04 MB 2025-02-15 06:38:40,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 06:38:40,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43715.35 MB 2025-02-15 06:38:40,326 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 06:38:40,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:40,328 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:38:40,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:40,329 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:38:40,333 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:38:40,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:40,335 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:38:40,335 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:38:56,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:56,683 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:38:56,688 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:38:56,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:56,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:38:56,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:38:56,696 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:39:14,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:39:14,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:39:14,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.78 seconds 2025-02-15 06:39:14,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:14,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33840.55 MB 2025-02-15 06:39:14,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37882.02 MB 2025-02-15 06:39:14,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-15 06:39:14,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66228.06 MB 2025-02-15 06:39:14,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43792.73 MB 2025-02-15 06:39:14,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22435.33 MB 2025-02-15 06:39:14,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46709.30 MB 2025-02-15 06:39:14,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:39:14,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:39:14,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:39:14,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:14,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37882.02 MB 2025-02-15 06:39:14,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34628.95 MB 2025-02-15 06:39:14,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3253.07 MB 2025-02-15 06:39:14,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43792.73 MB 2025-02-15 06:39:14,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53196.36 MB 2025-02-15 06:39:14,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9403.63 MB 2025-02-15 06:39:14,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50093.48 MB 2025-02-15 06:39:16,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:39:16,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:39:16,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:39:16,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34628.95 MB 2025-02-15 06:39:16,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35159.80 MB 2025-02-15 06:39:16,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:39:16,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53196.36 MB 2025-02-15 06:39:16,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41165.00 MB 2025-02-15 06:39:16,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12031.36 MB 2025-02-15 06:39:16,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39138.34 MB 2025-02-15 06:39:16,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:39:16,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:39:16,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:39:16,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35159.80 MB 2025-02-15 06:39:16,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37049.33 MB 2025-02-15 06:39:16,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:39:16,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:39:16,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42108.72 MB 2025-02-15 06:39:16,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:39:16,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38466.76 MB 2025-02-15 06:39:16,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:39:16,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:39:16,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:39:16,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37049.33 MB 2025-02-15 06:39:16,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39291.19 MB 2025-02-15 06:39:16,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:39:16,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42108.72 MB 2025-02-15 06:39:16,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:39:16,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:39:16,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44835.47 MB 2025-02-15 06:39:16,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:39:16,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:39:16,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:39:16,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35159.80 MB 2025-02-15 06:39:16,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39291.19 MB 2025-02-15 06:39:16,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:39:16,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:39:16,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:39:16,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:39:16,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44835.47 MB 2025-02-15 06:39:16,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:39:16,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:39:16,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:39:16,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40824.73 MB 2025-02-15 06:39:16,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41591.73 MB 2025-02-15 06:39:16,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:39:16,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47771.03 MB 2025-02-15 06:39:16,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 06:39:16,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 06:39:16,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42299.52 MB 2025-02-15 06:39:16,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:39:16,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:39:16,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:39:16,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42004.62 MB 2025-02-15 06:39:16,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42230.94 MB 2025-02-15 06:39:16,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.32 MB 2025-02-15 06:39:16,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 06:39:16,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 06:39:16,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:39:16,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42470.54 MB 2025-02-15 06:39:16,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:39:16,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:39:16,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.23 seconds 2025-02-15 06:39:16,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:16,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29861.73 MB 2025-02-15 06:39:16,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42431.79 MB 2025-02-15 06:39:16,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12570.06 MB 2025-02-15 06:39:16,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66228.06 MB 2025-02-15 06:39:16,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 06:39:16,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18045.99 MB 2025-02-15 06:39:16,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42470.54 MB 2025-02-15 06:39:17,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:39:17,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:39:17,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:39:17,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:17,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42431.79 MB 2025-02-15 06:39:17,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34848.28 MB 2025-02-15 06:39:17,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7583.51 MB 2025-02-15 06:39:17,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 06:39:17,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 06:39:17,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:39:17,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44928.40 MB 2025-02-15 06:39:17,213 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 06:39:17,213 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:39:17,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:39:17,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:39:17,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:39:17,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:39:17,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34848.28 MB 2025-02-15 06:39:17,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43236.70 MB 2025-02-15 06:39:17,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 06:39:17,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 06:39:17,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58609.11 MB 2025-02-15 06:39:17,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10427.04 MB 2025-02-15 06:39:17,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43236.70 MB 2025-02-15 06:39:17,376 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 06:39:17,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:39:17,378 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:39:17,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:39:17,379 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:39:17,383 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:39:17,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:39:17,384 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:39:17,384 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:40:11,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:11,237 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:40:11,242 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:40:11,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:11,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 366, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:40:11,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:11,246 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 366, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:40:16,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:40:16,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:40:16,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.65 seconds 2025-02-15 06:40:16,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:16,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28433.25 MB 2025-02-15 06:40:16,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29728.51 MB 2025-02-15 06:40:16,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1295.25 MB 2025-02-15 06:40:16,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71118.62 MB 2025-02-15 06:40:16,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38388.37 MB 2025-02-15 06:40:16,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32730.25 MB 2025-02-15 06:40:16,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38584.10 MB 2025-02-15 06:40:16,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:40:16,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:40:16,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:40:16,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:16,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29728.51 MB 2025-02-15 06:40:16,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30075.07 MB 2025-02-15 06:40:16,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 346.56 MB 2025-02-15 06:40:16,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38388.37 MB 2025-02-15 06:40:16,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38388.37 MB 2025-02-15 06:40:16,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:40:16,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34307.52 MB 2025-02-15 06:40:18,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:40:18,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:40:18,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.56 seconds 2025-02-15 06:40:18,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30075.07 MB 2025-02-15 06:40:18,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30507.70 MB 2025-02-15 06:40:18,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.64 MB 2025-02-15 06:40:18,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38388.37 MB 2025-02-15 06:40:18,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37444.65 MB 2025-02-15 06:40:18,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 06:40:18,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34499.52 MB 2025-02-15 06:40:18,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:40:18,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:40:18,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:40:18,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30507.70 MB 2025-02-15 06:40:18,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32047.52 MB 2025-02-15 06:40:18,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1539.82 MB 2025-02-15 06:40:18,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-15 06:40:18,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37444.65 MB 2025-02-15 06:40:18,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:40:18,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33202.73 MB 2025-02-15 06:40:18,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:40:18,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:40:18,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:40:18,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32047.52 MB 2025-02-15 06:40:18,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33875.17 MB 2025-02-15 06:40:18,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.65 MB 2025-02-15 06:40:18,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-15 06:40:18,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42062.58 MB 2025-02-15 06:40:18,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4617.93 MB 2025-02-15 06:40:18,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38395.85 MB 2025-02-15 06:40:18,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:40:18,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:40:18,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 06:40:18,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30507.70 MB 2025-02-15 06:40:18,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33875.17 MB 2025-02-15 06:40:18,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3367.47 MB 2025-02-15 06:40:18,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-15 06:40:18,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42062.58 MB 2025-02-15 06:40:18,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4617.93 MB 2025-02-15 06:40:18,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38395.85 MB 2025-02-15 06:40:18,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:40:18,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:40:18,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 06:40:18,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35125.01 MB 2025-02-15 06:40:18,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35750.12 MB 2025-02-15 06:40:18,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.11 MB 2025-02-15 06:40:18,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42062.58 MB 2025-02-15 06:40:18,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42396.02 MB 2025-02-15 06:40:18,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 333.45 MB 2025-02-15 06:40:18,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36326.96 MB 2025-02-15 06:40:18,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:40:18,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:40:18,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:40:18,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36086.62 MB 2025-02-15 06:40:18,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36305.03 MB 2025-02-15 06:40:18,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.41 MB 2025-02-15 06:40:18,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42396.02 MB 2025-02-15 06:40:18,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42396.02 MB 2025-02-15 06:40:18,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:40:18,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36421.26 MB 2025-02-15 06:40:18,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:40:18,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:40:18,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.57 seconds 2025-02-15 06:40:18,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:18,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27158.08 MB 2025-02-15 06:40:18,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36506.10 MB 2025-02-15 06:40:18,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9348.02 MB 2025-02-15 06:40:18,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71118.62 MB 2025-02-15 06:40:18,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42396.02 MB 2025-02-15 06:40:18,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28722.59 MB 2025-02-15 06:40:18,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36506.10 MB 2025-02-15 06:40:19,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:40:19,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:40:19,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:40:19,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:19,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36506.10 MB 2025-02-15 06:40:19,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39520.14 MB 2025-02-15 06:40:19,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 06:40:19,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42396.02 MB 2025-02-15 06:40:19,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42396.02 MB 2025-02-15 06:40:19,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:40:19,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39821.50 MB 2025-02-15 06:40:19,105 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:40:19,105 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-15 06:40:19,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:40:19,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:40:19,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:40:19,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:40:19,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31813.24 MB 2025-02-15 06:40:19,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40252.27 MB 2025-02-15 06:40:19,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:40:19,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42396.02 MB 2025-02-15 06:40:19,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50786.73 MB 2025-02-15 06:40:19,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:40:19,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40252.27 MB 2025-02-15 06:40:19,270 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:40:19,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:19,271 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:40:19,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:19,272 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:40:19,277 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:40:19,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:40:19,278 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:40:19,278 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-15 06:41:07,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:07,111 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:41:07,119 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:41:07,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:07,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:41:07,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:07,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:41:25,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:41:25,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:41:25,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.91 seconds 2025-02-15 06:41:25,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:25,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33945.07 MB 2025-02-15 06:41:25,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38039.63 MB 2025-02-15 06:41:25,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4094.56 MB 2025-02-15 06:41:25,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63371.74 MB 2025-02-15 06:41:25,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43845.16 MB 2025-02-15 06:41:25,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19526.58 MB 2025-02-15 06:41:25,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47041.13 MB 2025-02-15 06:41:25,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:41:25,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:41:25,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:41:25,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:25,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38039.63 MB 2025-02-15 06:41:25,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34707.98 MB 2025-02-15 06:41:25,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3331.64 MB 2025-02-15 06:41:25,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43845.16 MB 2025-02-15 06:41:25,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54098.13 MB 2025-02-15 06:41:25,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10252.98 MB 2025-02-15 06:41:25,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50391.21 MB 2025-02-15 06:41:27,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:41:27,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:41:27,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 06:41:27,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34707.98 MB 2025-02-15 06:41:27,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35238.83 MB 2025-02-15 06:41:27,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:41:27,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54098.13 MB 2025-02-15 06:41:27,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41873.83 MB 2025-02-15 06:41:27,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12224.30 MB 2025-02-15 06:41:27,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39217.37 MB 2025-02-15 06:41:27,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:41:27,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:41:27,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:41:27,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35238.83 MB 2025-02-15 06:41:27,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37128.36 MB 2025-02-15 06:41:27,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:41:27,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41873.83 MB 2025-02-15 06:41:27,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41873.83 MB 2025-02-15 06:41:27,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:41:27,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38545.79 MB 2025-02-15 06:41:27,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:41:27,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:41:27,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:41:27,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37128.36 MB 2025-02-15 06:41:27,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39370.22 MB 2025-02-15 06:41:27,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:41:27,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41873.83 MB 2025-02-15 06:41:27,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47536.14 MB 2025-02-15 06:41:27,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:41:27,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44914.50 MB 2025-02-15 06:41:27,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:41:27,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:41:27,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:41:27,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35238.83 MB 2025-02-15 06:41:27,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39370.22 MB 2025-02-15 06:41:27,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:41:27,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41873.83 MB 2025-02-15 06:41:27,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47536.14 MB 2025-02-15 06:41:27,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:41:27,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44914.50 MB 2025-02-15 06:41:27,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:41:27,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:41:27,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:41:27,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40903.76 MB 2025-02-15 06:41:27,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41670.76 MB 2025-02-15 06:41:27,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:41:27,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47536.14 MB 2025-02-15 06:41:27,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47949.28 MB 2025-02-15 06:41:27,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:41:27,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42378.55 MB 2025-02-15 06:41:27,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:41:27,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:41:27,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:41:27,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42083.65 MB 2025-02-15 06:41:27,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42311.58 MB 2025-02-15 06:41:27,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.93 MB 2025-02-15 06:41:27,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47949.28 MB 2025-02-15 06:41:27,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47949.28 MB 2025-02-15 06:41:27,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:41:27,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42556.68 MB 2025-02-15 06:41:27,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:41:27,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:41:27,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.38 seconds 2025-02-15 06:41:27,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29913.99 MB 2025-02-15 06:41:27,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42511.96 MB 2025-02-15 06:41:27,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12597.97 MB 2025-02-15 06:41:27,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63371.74 MB 2025-02-15 06:41:27,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47949.28 MB 2025-02-15 06:41:27,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15422.46 MB 2025-02-15 06:41:27,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42556.68 MB 2025-02-15 06:41:27,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:41:27,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:41:27,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:41:27,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42511.96 MB 2025-02-15 06:41:27,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34907.64 MB 2025-02-15 06:41:27,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7604.32 MB 2025-02-15 06:41:27,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47949.28 MB 2025-02-15 06:41:27,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47949.28 MB 2025-02-15 06:41:27,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:41:27,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45015.49 MB 2025-02-15 06:41:27,795 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-15 06:41:27,795 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:41:27,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:41:27,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:41:27,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:41:27,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:27,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34907.64 MB 2025-02-15 06:41:27,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43317.45 MB 2025-02-15 06:41:27,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-15 06:41:27,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47949.28 MB 2025-02-15 06:41:27,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56310.63 MB 2025-02-15 06:41:27,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-15 06:41:27,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43317.45 MB 2025-02-15 06:41:27,960 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-15 06:41:27,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:27,961 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:41:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:41:27,967 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:41:27,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:27,968 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:41:27,968 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:41:37,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:37,150 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:41:37,157 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:41:37,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:37,163 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1127, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:41:37,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:37,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1127, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:41:54,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:41:54,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:41:54,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.67 seconds 2025-02-15 06:41:54,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:54,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33736.02 MB 2025-02-15 06:41:54,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37724.81 MB 2025-02-15 06:41:54,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3988.78 MB 2025-02-15 06:41:54,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68851.60 MB 2025-02-15 06:41:54,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43738.20 MB 2025-02-15 06:41:54,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25113.40 MB 2025-02-15 06:41:54,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46604.78 MB 2025-02-15 06:41:54,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:41:54,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:41:54,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:41:54,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:54,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37724.81 MB 2025-02-15 06:41:54,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34550.97 MB 2025-02-15 06:41:54,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3173.83 MB 2025-02-15 06:41:54,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43738.20 MB 2025-02-15 06:41:54,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52883.88 MB 2025-02-15 06:41:54,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9145.68 MB 2025-02-15 06:41:54,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49674.92 MB 2025-02-15 06:41:56,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:41:56,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:41:56,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:41:56,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:56,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34550.97 MB 2025-02-15 06:41:56,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35081.82 MB 2025-02-15 06:41:56,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:41:56,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52883.88 MB 2025-02-15 06:41:56,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41165.00 MB 2025-02-15 06:41:56,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11718.89 MB 2025-02-15 06:41:56,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39060.36 MB 2025-02-15 06:41:56,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:41:56,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:41:56,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:41:56,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:56,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35081.82 MB 2025-02-15 06:41:56,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36971.35 MB 2025-02-15 06:41:56,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:41:56,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:41:56,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42108.72 MB 2025-02-15 06:41:56,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:41:56,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38388.78 MB 2025-02-15 06:41:57,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:41:57,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:41:57,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:41:57,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36971.35 MB 2025-02-15 06:41:57,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39213.21 MB 2025-02-15 06:41:57,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:41:57,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42108.72 MB 2025-02-15 06:41:57,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:41:57,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:41:57,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44757.49 MB 2025-02-15 06:41:57,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:41:57,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:41:57,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:41:57,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35081.82 MB 2025-02-15 06:41:57,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39213.21 MB 2025-02-15 06:41:57,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:41:57,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:41:57,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:41:57,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:41:57,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44757.49 MB 2025-02-15 06:41:57,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:41:57,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:41:57,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:41:57,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40746.75 MB 2025-02-15 06:41:57,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41513.75 MB 2025-02-15 06:41:57,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:41:57,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47771.03 MB 2025-02-15 06:41:57,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:41:57,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:41:57,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42221.54 MB 2025-02-15 06:41:57,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:41:57,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:41:57,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:41:57,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41926.64 MB 2025-02-15 06:41:57,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42155.04 MB 2025-02-15 06:41:57,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-15 06:41:57,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:41:57,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:41:57,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:41:57,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42372.49 MB 2025-02-15 06:41:57,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:41:57,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:41:57,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.09 seconds 2025-02-15 06:41:57,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29809.47 MB 2025-02-15 06:41:57,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42356.12 MB 2025-02-15 06:41:57,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12546.65 MB 2025-02-15 06:41:57,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68851.60 MB 2025-02-15 06:41:57,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:41:57,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20667.43 MB 2025-02-15 06:41:57,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42372.49 MB 2025-02-15 06:41:57,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:41:57,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:41:57,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:41:57,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42356.12 MB 2025-02-15 06:41:57,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34813.32 MB 2025-02-15 06:41:57,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7542.80 MB 2025-02-15 06:41:57,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:41:57,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:41:57,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:41:57,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44867.78 MB 2025-02-15 06:41:57,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:41:57,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:41:57,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:41:57,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:41:57,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:41:57,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:41:57,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34813.32 MB 2025-02-15 06:41:57,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43252.34 MB 2025-02-15 06:41:57,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:41:57,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:41:57,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56574.87 MB 2025-02-15 06:41:57,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:41:57,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43252.34 MB 2025-02-15 06:41:57,709 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:41:57,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:57,710 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:41:57,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:57,711 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:41:57,716 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:41:57,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:41:57,717 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:41:57,717 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:42:09,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:09,440 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:42:09,445 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:42:09,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:09,448 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:42:09,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:09,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:42:11,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:42:11,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:42:11,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-15 06:42:11,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:11,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26949.04 MB 2025-02-15 06:42:11,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27490.50 MB 2025-02-15 06:42:11,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 06:42:11,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69159.88 MB 2025-02-15 06:42:11,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:11,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29408.36 MB 2025-02-15 06:42:11,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36420.41 MB 2025-02-15 06:42:11,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:42:11,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:42:11,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:42:11,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:11,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27490.50 MB 2025-02-15 06:42:11,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27696.65 MB 2025-02-15 06:42:11,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.15 MB 2025-02-15 06:42:11,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:11,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:11,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:11,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29527.24 MB 2025-02-15 06:42:12,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:42:12,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:42:12,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 06:42:12,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27696.65 MB 2025-02-15 06:42:12,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27889.08 MB 2025-02-15 06:42:12,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 06:42:12,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:12,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:12,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31866.30 MB 2025-02-15 06:42:12,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:42:12,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:42:12,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:42:12,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27889.01 MB 2025-02-15 06:42:12,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28573.80 MB 2025-02-15 06:42:12,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 06:42:12,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:12,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:12,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29087.63 MB 2025-02-15 06:42:12,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:42:12,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:42:12,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:42:12,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28573.80 MB 2025-02-15 06:42:12,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29386.52 MB 2025-02-15 06:42:12,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 06:42:12,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:12,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:12,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31396.28 MB 2025-02-15 06:42:12,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:42:12,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:42:12,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:42:12,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27889.01 MB 2025-02-15 06:42:12,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29386.52 MB 2025-02-15 06:42:12,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 06:42:12,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:12,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 06:42:12,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31396.28 MB 2025-02-15 06:42:12,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:42:12,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:42:12,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 06:42:12,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29942.43 MB 2025-02-15 06:42:12,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30220.46 MB 2025-02-15 06:42:12,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 06:42:12,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 06:42:12,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39896.22 MB 2025-02-15 06:42:12,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-15 06:42:12,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30486.87 MB 2025-02-15 06:42:12,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:42:12,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:42:12,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:42:12,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30370.14 MB 2025-02-15 06:42:12,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30593.42 MB 2025-02-15 06:42:12,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.27 MB 2025-02-15 06:42:12,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39896.22 MB 2025-02-15 06:42:12,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39896.22 MB 2025-02-15 06:42:12,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30604.20 MB 2025-02-15 06:42:12,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:42:12,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:42:12,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-15 06:42:12,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26415.97 MB 2025-02-15 06:42:12,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30794.34 MB 2025-02-15 06:42:12,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4378.37 MB 2025-02-15 06:42:12,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69159.88 MB 2025-02-15 06:42:12,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39896.22 MB 2025-02-15 06:42:12,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29263.66 MB 2025-02-15 06:42:12,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30794.34 MB 2025-02-15 06:42:12,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:42:12,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:42:12,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:42:12,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:12,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30794.34 MB 2025-02-15 06:42:12,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30214.12 MB 2025-02-15 06:42:12,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -580.22 MB 2025-02-15 06:42:12,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39896.22 MB 2025-02-15 06:42:12,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39896.22 MB 2025-02-15 06:42:12,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:12,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31898.66 MB 2025-02-15 06:42:13,015 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 06:42:13,015 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:42:13,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:42:13,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:42:13,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:42:13,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:13,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30214.12 MB 2025-02-15 06:42:13,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38647.41 MB 2025-02-15 06:42:13,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 06:42:13,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39896.22 MB 2025-02-15 06:42:13,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48280.63 MB 2025-02-15 06:42:13,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 06:42:13,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38647.41 MB 2025-02-15 06:42:13,185 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 06:42:13,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:13,187 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:42:13,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:13,187 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:42:13,192 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:42:13,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:13,193 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:42:13,193 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:42:20,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:20,511 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:42:20,516 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:42:20,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:20,520 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 196, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:42:20,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:20,521 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 196, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:42:23,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:42:23,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:42:23,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.12 seconds 2025-02-15 06:42:23,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:23,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27248.67 MB 2025-02-15 06:42:23,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27942.30 MB 2025-02-15 06:42:23,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 693.63 MB 2025-02-15 06:42:23,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56665.05 MB 2025-02-15 06:42:23,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:23,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21109.93 MB 2025-02-15 06:42:23,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36946.53 MB 2025-02-15 06:42:23,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:42:23,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:42:23,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:42:23,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:23,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27942.30 MB 2025-02-15 06:42:23,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28151.88 MB 2025-02-15 06:42:23,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.58 MB 2025-02-15 06:42:23,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:23,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:23,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:23,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30442.50 MB 2025-02-15 06:42:24,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:42:24,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:42:24,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-15 06:42:24,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28151.88 MB 2025-02-15 06:42:24,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28388.11 MB 2025-02-15 06:42:24,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-15 06:42:24,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:24,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:24,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:24,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32321.53 MB 2025-02-15 06:42:24,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:42:24,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:42:24,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:42:24,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28388.11 MB 2025-02-15 06:42:24,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29228.75 MB 2025-02-15 06:42:24,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-15 06:42:24,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:24,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:24,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:24,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29859.51 MB 2025-02-15 06:42:24,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:42:24,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:42:24,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:42:24,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29228.75 MB 2025-02-15 06:42:24,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30226.41 MB 2025-02-15 06:42:24,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-15 06:42:24,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:24,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:24,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:24,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32693.58 MB 2025-02-15 06:42:24,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:42:24,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:42:24,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:42:24,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28388.11 MB 2025-02-15 06:42:24,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30226.41 MB 2025-02-15 06:42:24,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-15 06:42:24,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:24,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:42:24,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:24,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32693.58 MB 2025-02-15 06:42:24,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:42:24,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:42:24,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:42:24,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30908.84 MB 2025-02-15 06:42:24,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31250.15 MB 2025-02-15 06:42:24,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.32 MB 2025-02-15 06:42:24,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:42:24,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35735.47 MB 2025-02-15 06:42:24,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 06:42:24,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31571.37 MB 2025-02-15 06:42:24,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:42:24,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:42:24,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:42:24,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31433.90 MB 2025-02-15 06:42:24,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31638.10 MB 2025-02-15 06:42:24,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.21 MB 2025-02-15 06:42:24,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35735.47 MB 2025-02-15 06:42:24,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35739.66 MB 2025-02-15 06:42:24,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 06:42:24,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31672.17 MB 2025-02-15 06:42:24,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:42:24,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:42:24,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.20 seconds 2025-02-15 06:42:24,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.79 MB 2025-02-15 06:42:24,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31839.18 MB 2025-02-15 06:42:24,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5273.39 MB 2025-02-15 06:42:24,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56665.05 MB 2025-02-15 06:42:24,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35739.66 MB 2025-02-15 06:42:24,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20925.38 MB 2025-02-15 06:42:24,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.18 MB 2025-02-15 06:42:24,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:42:24,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:42:24,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:42:24,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:24,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31839.18 MB 2025-02-15 06:42:24,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30521.95 MB 2025-02-15 06:42:24,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1317.22 MB 2025-02-15 06:42:24,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35739.66 MB 2025-02-15 06:42:24,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35739.66 MB 2025-02-15 06:42:24,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:42:24,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32040.14 MB 2025-02-15 06:42:25,014 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:42:25,014 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:42:25,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:42:25,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:42:25,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:42:25,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:42:25,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30521.95 MB 2025-02-15 06:42:25,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38960.98 MB 2025-02-15 06:42:25,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:42:25,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35739.66 MB 2025-02-15 06:42:25,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44130.37 MB 2025-02-15 06:42:25,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:42:25,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38960.98 MB 2025-02-15 06:42:25,181 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:42:25,183 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:25,183 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:42:25,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:25,184 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:42:25,188 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:42:25,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:42:25,189 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:42:25,189 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:43:14,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:14,389 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:43:14,394 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:43:14,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:14,398 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:43:14,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:14,399 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:43:16,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:43:16,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:43:16,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.28 seconds 2025-02-15 06:43:16,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:16,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26907.23 MB 2025-02-15 06:43:16,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27427.45 MB 2025-02-15 06:43:16,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 520.22 MB 2025-02-15 06:43:16,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56715.38 MB 2025-02-15 06:43:16,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:16,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21160.26 MB 2025-02-15 06:43:16,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36378.60 MB 2025-02-15 06:43:16,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:43:16,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:43:16,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:43:16,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:16,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27427.45 MB 2025-02-15 06:43:16,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27665.46 MB 2025-02-15 06:43:16,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 238.00 MB 2025-02-15 06:43:16,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:16,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:16,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:16,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29464.19 MB 2025-02-15 06:43:17,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:43:17,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:43:17,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 06:43:17,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27665.46 MB 2025-02-15 06:43:17,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27857.89 MB 2025-02-15 06:43:17,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 06:43:17,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:17,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:17,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31835.11 MB 2025-02-15 06:43:17,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:43:17,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:43:17,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:43:17,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27857.82 MB 2025-02-15 06:43:17,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28542.61 MB 2025-02-15 06:43:17,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 06:43:17,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:17,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:17,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29056.43 MB 2025-02-15 06:43:17,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:43:17,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:43:17,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:43:17,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28542.61 MB 2025-02-15 06:43:17,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29355.32 MB 2025-02-15 06:43:17,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 06:43:17,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:17,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:17,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31365.09 MB 2025-02-15 06:43:17,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:43:17,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:43:17,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:43:17,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27857.82 MB 2025-02-15 06:43:17,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29355.32 MB 2025-02-15 06:43:17,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 06:43:17,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:17,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:43:17,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31365.09 MB 2025-02-15 06:43:17,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:43:17,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:43:17,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 06:43:17,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29911.23 MB 2025-02-15 06:43:17,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30189.27 MB 2025-02-15 06:43:17,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 06:43:17,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:43:17,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35701.92 MB 2025-02-15 06:43:17,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-15 06:43:17,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30455.09 MB 2025-02-15 06:43:17,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:43:17,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:43:17,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:43:17,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30338.95 MB 2025-02-15 06:43:17,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30566.58 MB 2025-02-15 06:43:17,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.63 MB 2025-02-15 06:43:17,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35701.92 MB 2025-02-15 06:43:17,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35701.92 MB 2025-02-15 06:43:17,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30581.54 MB 2025-02-15 06:43:17,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:43:17,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:43:17,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.16 seconds 2025-02-15 06:43:17,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26395.07 MB 2025-02-15 06:43:17,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30767.58 MB 2025-02-15 06:43:17,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4372.51 MB 2025-02-15 06:43:17,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56715.38 MB 2025-02-15 06:43:17,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35701.92 MB 2025-02-15 06:43:17,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21013.46 MB 2025-02-15 06:43:17,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30767.58 MB 2025-02-15 06:43:17,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:43:17,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:43:17,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:43:17,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30767.58 MB 2025-02-15 06:43:17,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30194.35 MB 2025-02-15 06:43:17,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -573.22 MB 2025-02-15 06:43:17,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35701.92 MB 2025-02-15 06:43:17,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35701.92 MB 2025-02-15 06:43:17,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:43:17,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31872.30 MB 2025-02-15 06:43:17,850 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 06:43:17,850 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:43:17,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:43:17,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:43:17,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:43:17,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:43:17,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30194.35 MB 2025-02-15 06:43:17,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38629.95 MB 2025-02-15 06:43:17,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 06:43:17,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35701.92 MB 2025-02-15 06:43:17,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44090.52 MB 2025-02-15 06:43:17,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 06:43:17,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38629.95 MB 2025-02-15 06:43:18,016 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 06:43:18,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:18,018 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:43:18,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:18,019 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:43:18,023 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:43:18,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:43:18,025 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:43:18,025 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:44:29,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:29,544 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:44:29,549 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:44:29,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:29,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:44:29,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:29,554 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:44:46,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:44:46,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:44:46,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.40 seconds 2025-02-15 06:44:46,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:46,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33798.74 MB 2025-02-15 06:44:46,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37818.98 MB 2025-02-15 06:44:46,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-15 06:44:46,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52479.13 MB 2025-02-15 06:44:46,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43769.66 MB 2025-02-15 06:44:46,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8709.47 MB 2025-02-15 06:44:46,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46668.30 MB 2025-02-15 06:44:47,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:44:47,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:44:47,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:44:47,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:47,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37818.98 MB 2025-02-15 06:44:47,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34597.76 MB 2025-02-15 06:44:47,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3221.22 MB 2025-02-15 06:44:47,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43769.66 MB 2025-02-15 06:44:47,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53164.90 MB 2025-02-15 06:44:47,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9395.24 MB 2025-02-15 06:44:47,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50023.43 MB 2025-02-15 06:44:48,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:44:48,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:44:48,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 06:44:48,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:48,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34597.76 MB 2025-02-15 06:44:48,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35128.60 MB 2025-02-15 06:44:48,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:44:48,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53164.90 MB 2025-02-15 06:44:48,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41165.00 MB 2025-02-15 06:44:48,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11999.90 MB 2025-02-15 06:44:48,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39107.15 MB 2025-02-15 06:44:48,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:44:48,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:44:48,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:44:48,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:48,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35128.60 MB 2025-02-15 06:44:48,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37018.14 MB 2025-02-15 06:44:48,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:44:48,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:44:48,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42108.72 MB 2025-02-15 06:44:48,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:44:48,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38435.57 MB 2025-02-15 06:44:49,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:44:49,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:44:49,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:44:49,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37018.14 MB 2025-02-15 06:44:49,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39259.99 MB 2025-02-15 06:44:49,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:44:49,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42108.72 MB 2025-02-15 06:44:49,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:44:49,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:44:49,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44804.28 MB 2025-02-15 06:44:49,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:44:49,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:44:49,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:44:49,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35128.60 MB 2025-02-15 06:44:49,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39259.99 MB 2025-02-15 06:44:49,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:44:49,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41165.00 MB 2025-02-15 06:44:49,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47771.03 MB 2025-02-15 06:44:49,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:44:49,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44804.28 MB 2025-02-15 06:44:49,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:44:49,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:44:49,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:44:49,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40793.54 MB 2025-02-15 06:44:49,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41560.54 MB 2025-02-15 06:44:49,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:44:49,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47771.03 MB 2025-02-15 06:44:49,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:44:49,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:44:49,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42268.33 MB 2025-02-15 06:44:49,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:44:49,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:44:49,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:44:49,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41973.43 MB 2025-02-15 06:44:49,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42201.63 MB 2025-02-15 06:44:49,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-15 06:44:49,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:44:49,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:44:49,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:44:49,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42434.51 MB 2025-02-15 06:44:49,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:44:49,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:44:49,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.80 seconds 2025-02-15 06:44:49,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29840.82 MB 2025-02-15 06:44:49,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42401.64 MB 2025-02-15 06:44:49,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12560.82 MB 2025-02-15 06:44:49,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52479.13 MB 2025-02-15 06:44:49,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:44:49,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4294.97 MB 2025-02-15 06:44:49,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42434.51 MB 2025-02-15 06:44:49,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:44:49,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:44:49,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 06:44:49,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42401.64 MB 2025-02-15 06:44:49,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34829.37 MB 2025-02-15 06:44:49,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7572.27 MB 2025-02-15 06:44:49,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:44:49,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48184.16 MB 2025-02-15 06:44:49,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:44:49,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44900.94 MB 2025-02-15 06:44:49,644 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 06:44:49,644 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:44:49,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:44:49,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:44:49,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:44:49,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:44:49,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34829.37 MB 2025-02-15 06:44:49,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43224.58 MB 2025-02-15 06:44:49,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 06:44:49,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48184.16 MB 2025-02-15 06:44:49,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56530.83 MB 2025-02-15 06:44:49,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 06:44:49,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43224.58 MB 2025-02-15 06:44:49,810 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 06:44:49,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:49,811 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:44:49,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:49,812 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:44:49,817 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:44:49,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:44:49,819 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:44:49,819 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:45:46,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:45:46,904 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:45:46,909 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:45:46,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:45:46,913 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1606, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:45:46,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:45:46,914 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1606, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:46:11,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:46:11,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:46:11,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.79 seconds 2025-02-15 06:46:11,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:11,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37073.77 MB 2025-02-15 06:46:11,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42757.32 MB 2025-02-15 06:46:11,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5683.54 MB 2025-02-15 06:46:11,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64877.49 MB 2025-02-15 06:46:11,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53764.69 MB 2025-02-15 06:46:11,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11112.81 MB 2025-02-15 06:46:11,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51754.47 MB 2025-02-15 06:46:11,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:46:11,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:46:11,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:46:11,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:11,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42757.32 MB 2025-02-15 06:46:11,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37041.15 MB 2025-02-15 06:46:11,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5716.17 MB 2025-02-15 06:46:11,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53764.69 MB 2025-02-15 06:46:11,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64133.01 MB 2025-02-15 06:46:11,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10368.32 MB 2025-02-15 06:46:11,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58349.95 MB 2025-02-15 06:46:13,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:46:13,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:46:13,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:46:13,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:13,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37041.15 MB 2025-02-15 06:46:13,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37571.99 MB 2025-02-15 06:46:13,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:46:13,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64133.01 MB 2025-02-15 06:46:13,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48079.31 MB 2025-02-15 06:46:13,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16053.70 MB 2025-02-15 06:46:13,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41550.54 MB 2025-02-15 06:46:13,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:46:13,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:46:13,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:46:13,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:13,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37571.99 MB 2025-02-15 06:46:13,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39461.52 MB 2025-02-15 06:46:13,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:46:13,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48079.31 MB 2025-02-15 06:46:13,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48079.31 MB 2025-02-15 06:46:13,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:46:13,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40878.95 MB 2025-02-15 06:46:13,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:46:13,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:46:13,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:46:13,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:13,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39461.52 MB 2025-02-15 06:46:13,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41703.38 MB 2025-02-15 06:46:13,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:46:13,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48079.31 MB 2025-02-15 06:46:13,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50910.46 MB 2025-02-15 06:46:13,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:46:13,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47247.66 MB 2025-02-15 06:46:13,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:46:13,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:46:13,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:46:13,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:13,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37571.99 MB 2025-02-15 06:46:13,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41703.38 MB 2025-02-15 06:46:13,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:46:13,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48079.31 MB 2025-02-15 06:46:13,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50910.46 MB 2025-02-15 06:46:13,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:46:13,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47247.66 MB 2025-02-15 06:46:14,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:46:14,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:46:14,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:46:14,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:14,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43236.92 MB 2025-02-15 06:46:14,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44003.92 MB 2025-02-15 06:46:14,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:46:14,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50910.46 MB 2025-02-15 06:46:14,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51319.41 MB 2025-02-15 06:46:14,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 06:46:14,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44711.71 MB 2025-02-15 06:46:14,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:46:14,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:46:14,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:46:14,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:14,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44416.81 MB 2025-02-15 06:46:14,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44645.32 MB 2025-02-15 06:46:14,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.51 MB 2025-02-15 06:46:14,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51319.41 MB 2025-02-15 06:46:14,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51319.41 MB 2025-02-15 06:46:14,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:46:14,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44854.15 MB 2025-02-15 06:46:14,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:46:14,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:46:14,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.23 seconds 2025-02-15 06:46:14,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:14,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31478.34 MB 2025-02-15 06:46:14,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44846.17 MB 2025-02-15 06:46:14,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13367.83 MB 2025-02-15 06:46:14,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64877.49 MB 2025-02-15 06:46:14,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51319.41 MB 2025-02-15 06:46:14,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13558.09 MB 2025-02-15 06:46:14,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44854.15 MB 2025-02-15 06:46:14,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:46:14,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:46:14,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:46:14,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:14,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44846.17 MB 2025-02-15 06:46:14,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36478.76 MB 2025-02-15 06:46:14,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8367.41 MB 2025-02-15 06:46:14,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51319.41 MB 2025-02-15 06:46:14,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51319.41 MB 2025-02-15 06:46:14,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:46:14,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47355.07 MB 2025-02-15 06:46:14,434 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 06:46:14,435 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:46:14,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:46:14,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:46:14,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:46:14,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:46:14,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36478.76 MB 2025-02-15 06:46:14,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44909.16 MB 2025-02-15 06:46:14,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 06:46:14,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51319.41 MB 2025-02-15 06:46:14,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59699.63 MB 2025-02-15 06:46:14,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 06:46:14,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44909.16 MB 2025-02-15 06:46:14,602 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 06:46:14,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:46:14,603 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:46:14,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:46:14,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:46:14,609 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:46:14,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:46:14,610 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:46:14,610 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:47:13,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:13,895 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:47:13,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:47:13,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:13,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:47:13,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:13,907 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:47:32,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:47:32,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:47:32,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.79 seconds 2025-02-15 06:47:32,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:32,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34363.16 MB 2025-02-15 06:47:32,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38670.71 MB 2025-02-15 06:47:32,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 06:47:32,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68079.85 MB 2025-02-15 06:47:32,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48213.52 MB 2025-02-15 06:47:32,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19866.32 MB 2025-02-15 06:47:32,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47684.90 MB 2025-02-15 06:47:32,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:47:32,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:47:32,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:47:32,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:32,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38670.71 MB 2025-02-15 06:47:32,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35018.86 MB 2025-02-15 06:47:32,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-15 06:47:32,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48213.52 MB 2025-02-15 06:47:32,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56488.89 MB 2025-02-15 06:47:32,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8275.36 MB 2025-02-15 06:47:32,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51169.01 MB 2025-02-15 06:47:34,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:47:34,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:47:34,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:47:34,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:34,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35018.86 MB 2025-02-15 06:47:34,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35549.70 MB 2025-02-15 06:47:34,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:47:34,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56488.89 MB 2025-02-15 06:47:34,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39732.64 MB 2025-02-15 06:47:34,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16756.24 MB 2025-02-15 06:47:34,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39529.28 MB 2025-02-15 06:47:34,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:47:34,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:47:34,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:47:34,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:34,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35549.70 MB 2025-02-15 06:47:34,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37439.23 MB 2025-02-15 06:47:34,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:47:34,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39732.64 MB 2025-02-15 06:47:34,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42563.80 MB 2025-02-15 06:47:34,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:47:34,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38856.66 MB 2025-02-15 06:47:34,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:47:34,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:47:34,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:47:34,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:34,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37439.23 MB 2025-02-15 06:47:34,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39681.09 MB 2025-02-15 06:47:34,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:47:34,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42563.80 MB 2025-02-15 06:47:34,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48226.11 MB 2025-02-15 06:47:34,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:47:34,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45225.37 MB 2025-02-15 06:47:34,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:47:34,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:47:34,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:47:34,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:34,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35549.70 MB 2025-02-15 06:47:34,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39681.09 MB 2025-02-15 06:47:34,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:47:34,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39732.64 MB 2025-02-15 06:47:34,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48226.11 MB 2025-02-15 06:47:34,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 06:47:34,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45225.37 MB 2025-02-15 06:47:35,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:47:35,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:47:35,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:47:35,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:35,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41214.63 MB 2025-02-15 06:47:35,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41981.63 MB 2025-02-15 06:47:35,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:47:35,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48226.11 MB 2025-02-15 06:47:35,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48635.05 MB 2025-02-15 06:47:35,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 06:47:35,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42689.42 MB 2025-02-15 06:47:35,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:47:35,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:47:35,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:47:35,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:35,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42394.52 MB 2025-02-15 06:47:35,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42623.04 MB 2025-02-15 06:47:35,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-15 06:47:35,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48635.05 MB 2025-02-15 06:47:35,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48635.05 MB 2025-02-15 06:47:35,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:47:35,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42836.26 MB 2025-02-15 06:47:35,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:47:35,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:47:35,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.24 seconds 2025-02-15 06:47:35,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:35,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30123.03 MB 2025-02-15 06:47:35,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42824.01 MB 2025-02-15 06:47:35,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.98 MB 2025-02-15 06:47:35,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68079.85 MB 2025-02-15 06:47:35,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48635.05 MB 2025-02-15 06:47:35,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19444.79 MB 2025-02-15 06:47:35,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42836.26 MB 2025-02-15 06:47:35,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:47:35,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:47:35,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:47:35,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:35,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42824.01 MB 2025-02-15 06:47:35,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35125.36 MB 2025-02-15 06:47:35,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7698.65 MB 2025-02-15 06:47:35,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48635.05 MB 2025-02-15 06:47:35,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48635.05 MB 2025-02-15 06:47:35,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:47:35,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45334.45 MB 2025-02-15 06:47:35,435 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 06:47:35,436 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:47:35,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:47:35,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:47:35,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:47:35,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:47:35,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35125.36 MB 2025-02-15 06:47:35,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43560.21 MB 2025-02-15 06:47:35,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 06:47:35,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48635.05 MB 2025-02-15 06:47:35,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57021.56 MB 2025-02-15 06:47:35,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 06:47:35,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43560.21 MB 2025-02-15 06:47:35,604 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 06:47:35,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:35,606 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:47:35,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:35,607 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:47:35,611 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:47:35,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:35,612 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:47:35,612 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 06:47:46,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:46,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:47:46,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:47:46,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:46,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:47:46,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:47:46,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:48:05,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:48:05,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:48:05,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.82 seconds 2025-02-15 06:48:05,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:05,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34307.41 MB 2025-02-15 06:48:05,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38586.00 MB 2025-02-15 06:48:05,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-15 06:48:05,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69600.28 MB 2025-02-15 06:48:05,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48205.14 MB 2025-02-15 06:48:05,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21395.14 MB 2025-02-15 06:48:05,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47402.66 MB 2025-02-15 06:48:05,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:48:05,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:48:05,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:48:05,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:05,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38586.00 MB 2025-02-15 06:48:05,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34977.27 MB 2025-02-15 06:48:05,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3608.73 MB 2025-02-15 06:48:05,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48205.14 MB 2025-02-15 06:48:05,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56295.95 MB 2025-02-15 06:48:05,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8090.81 MB 2025-02-15 06:48:05,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50901.60 MB 2025-02-15 06:48:07,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:48:07,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:48:07,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:48:07,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34977.27 MB 2025-02-15 06:48:07,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35508.11 MB 2025-02-15 06:48:07,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:48:07,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56295.95 MB 2025-02-15 06:48:07,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43924.85 MB 2025-02-15 06:48:07,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12371.10 MB 2025-02-15 06:48:07,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39486.66 MB 2025-02-15 06:48:07,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:48:07,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:48:07,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:48:07,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35508.11 MB 2025-02-15 06:48:07,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37397.64 MB 2025-02-15 06:48:07,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:48:07,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:48:07,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43924.85 MB 2025-02-15 06:48:07,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:48:07,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38815.07 MB 2025-02-15 06:48:07,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:48:07,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:48:07,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:48:07,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37397.64 MB 2025-02-15 06:48:07,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39639.50 MB 2025-02-15 06:48:07,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:48:07,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:48:07,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48643.44 MB 2025-02-15 06:48:07,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 06:48:07,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45183.78 MB 2025-02-15 06:48:07,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:48:07,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:48:07,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:48:07,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35508.11 MB 2025-02-15 06:48:07,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39639.50 MB 2025-02-15 06:48:07,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:48:07,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:48:07,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48643.44 MB 2025-02-15 06:48:07,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 06:48:07,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45183.78 MB 2025-02-15 06:48:07,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:48:07,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:48:07,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:48:07,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41173.04 MB 2025-02-15 06:48:07,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41940.04 MB 2025-02-15 06:48:07,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:48:07,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48643.44 MB 2025-02-15 06:48:07,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49052.39 MB 2025-02-15 06:48:07,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 06:48:07,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42647.83 MB 2025-02-15 06:48:07,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:48:07,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:48:07,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:48:07,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42352.93 MB 2025-02-15 06:48:07,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42581.35 MB 2025-02-15 06:48:07,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.42 MB 2025-02-15 06:48:07,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49052.39 MB 2025-02-15 06:48:07,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49052.39 MB 2025-02-15 06:48:07,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:48:07,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42795.94 MB 2025-02-15 06:48:07,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:48:07,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:48:07,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.30 seconds 2025-02-15 06:48:07,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:07,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30095.16 MB 2025-02-15 06:48:07,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42782.23 MB 2025-02-15 06:48:07,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12687.07 MB 2025-02-15 06:48:07,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69600.28 MB 2025-02-15 06:48:07,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49052.39 MB 2025-02-15 06:48:07,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20547.90 MB 2025-02-15 06:48:07,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42795.94 MB 2025-02-15 06:48:08,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:48:08,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:48:08,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:48:08,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:08,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42782.23 MB 2025-02-15 06:48:08,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35095.96 MB 2025-02-15 06:48:08,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7686.27 MB 2025-02-15 06:48:08,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49052.39 MB 2025-02-15 06:48:08,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49052.39 MB 2025-02-15 06:48:08,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:48:08,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45291.44 MB 2025-02-15 06:48:08,147 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 06:48:08,147 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:48:08,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:48:08,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:48:08,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:48:08,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:48:08,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35095.96 MB 2025-02-15 06:48:08,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43526.64 MB 2025-02-15 06:48:08,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 06:48:08,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49052.39 MB 2025-02-15 06:48:08,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57434.70 MB 2025-02-15 06:48:08,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 06:48:08,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43526.64 MB 2025-02-15 06:48:08,315 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 06:48:08,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:48:08,316 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:48:08,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:48:08,317 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:48:08,322 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:48:08,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:48:08,323 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:48:08,323 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:49:34,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:34,736 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:49:34,741 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:49:34,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:34,745 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:49:34,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:34,746 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:49:37,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:49:37,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:49:37,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.94 seconds 2025-02-15 06:49:37,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:37,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27213.83 MB 2025-02-15 06:49:37,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27889.77 MB 2025-02-15 06:49:37,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-15 06:49:37,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70007.13 MB 2025-02-15 06:49:37,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:37,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34447.82 MB 2025-02-15 06:49:37,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36767.58 MB 2025-02-15 06:49:37,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:49:37,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:49:37,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:49:37,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:37,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27889.77 MB 2025-02-15 06:49:37,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28217.26 MB 2025-02-15 06:49:37,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 327.49 MB 2025-02-15 06:49:37,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:37,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:37,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:37,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30572.63 MB 2025-02-15 06:49:38,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:49:38,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:49:38,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-15 06:49:38,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28217.26 MB 2025-02-15 06:49:38,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28470.73 MB 2025-02-15 06:49:38,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 253.48 MB 2025-02-15 06:49:38,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:38,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:38,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:38,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32408.03 MB 2025-02-15 06:49:38,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:49:38,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:49:38,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:49:38,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28470.67 MB 2025-02-15 06:49:38,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29372.70 MB 2025-02-15 06:49:38,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.03 MB 2025-02-15 06:49:38,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:38,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:38,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:38,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30049.53 MB 2025-02-15 06:49:38,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:49:38,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:49:38,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:49:38,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29372.70 MB 2025-02-15 06:49:38,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30443.22 MB 2025-02-15 06:49:38,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1070.52 MB 2025-02-15 06:49:38,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:38,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:38,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:38,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33090.58 MB 2025-02-15 06:49:38,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:49:38,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:49:38,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 06:49:38,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28470.67 MB 2025-02-15 06:49:38,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30443.22 MB 2025-02-15 06:49:38,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1972.55 MB 2025-02-15 06:49:38,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:38,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 06:49:38,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:38,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33090.58 MB 2025-02-15 06:49:38,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:49:38,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:49:38,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:49:38,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31175.49 MB 2025-02-15 06:49:38,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31541.73 MB 2025-02-15 06:49:38,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 366.24 MB 2025-02-15 06:49:38,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 06:49:38,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35750.15 MB 2025-02-15 06:49:38,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-15 06:49:38,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31883.17 MB 2025-02-15 06:49:38,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:49:38,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:49:38,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:49:38,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31738.89 MB 2025-02-15 06:49:38,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31963.04 MB 2025-02-15 06:49:38,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.15 MB 2025-02-15 06:49:38,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35750.15 MB 2025-02-15 06:49:38,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35750.15 MB 2025-02-15 06:49:38,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:38,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32001.86 MB 2025-02-15 06:49:38,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:49:38,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:49:38,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.07 seconds 2025-02-15 06:49:38,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:38,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26548.37 MB 2025-02-15 06:49:38,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32163.94 MB 2025-02-15 06:49:38,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5615.58 MB 2025-02-15 06:49:38,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70007.13 MB 2025-02-15 06:49:38,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35750.15 MB 2025-02-15 06:49:38,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34256.98 MB 2025-02-15 06:49:38,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32163.94 MB 2025-02-15 06:49:39,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:49:39,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:49:39,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 06:49:39,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:39,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.94 MB 2025-02-15 06:49:39,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30563.22 MB 2025-02-15 06:49:39,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1600.73 MB 2025-02-15 06:49:39,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35750.15 MB 2025-02-15 06:49:39,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35750.15 MB 2025-02-15 06:49:39,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:49:39,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32163.95 MB 2025-02-15 06:49:39,099 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 06:49:39,099 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:49:39,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:49:39,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:49:39,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:49:39,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:49:39,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30563.22 MB 2025-02-15 06:49:39,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38994.68 MB 2025-02-15 06:49:39,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 06:49:39,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35750.15 MB 2025-02-15 06:49:39,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-15 06:49:39,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 06:49:39,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38994.68 MB 2025-02-15 06:49:39,265 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 06:49:39,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:39,267 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:49:39,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:39,268 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:49:39,272 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:49:39,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:49:39,273 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:49:39,273 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:50:38,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:50:38,971 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:50:38,976 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:50:38,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:50:38,980 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2057, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:50:38,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:50:38,981 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2057, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:51:10,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:51:10,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:51:10,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.74 seconds 2025-02-15 06:51:10,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:10,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40216.41 MB 2025-02-15 06:51:10,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47496.02 MB 2025-02-15 06:51:10,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7279.61 MB 2025-02-15 06:51:10,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52518.98 MB 2025-02-15 06:51:10,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55413.05 MB 2025-02-15 06:51:10,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2894.07 MB 2025-02-15 06:51:10,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56482.56 MB 2025-02-15 06:51:10,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:51:10,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:51:10,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 06:51:10,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:10,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47496.02 MB 2025-02-15 06:51:10,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39386.80 MB 2025-02-15 06:51:10,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8109.22 MB 2025-02-15 06:51:10,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55413.05 MB 2025-02-15 06:51:10,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71875.69 MB 2025-02-15 06:51:10,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16462.64 MB 2025-02-15 06:51:10,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69053.69 MB 2025-02-15 06:51:12,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:51:12,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:51:12,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:51:12,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:12,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39386.80 MB 2025-02-15 06:51:12,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39917.64 MB 2025-02-15 06:51:12,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:51:12,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71875.69 MB 2025-02-15 06:51:12,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46063.94 MB 2025-02-15 06:51:12,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25811.75 MB 2025-02-15 06:51:12,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43896.19 MB 2025-02-15 06:51:12,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:51:12,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:51:12,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:51:12,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:12,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39917.64 MB 2025-02-15 06:51:12,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41807.18 MB 2025-02-15 06:51:12,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:51:12,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46063.94 MB 2025-02-15 06:51:12,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47007.66 MB 2025-02-15 06:51:12,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 06:51:12,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43224.61 MB 2025-02-15 06:51:13,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:51:13,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:51:13,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:51:13,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41807.18 MB 2025-02-15 06:51:13,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44049.03 MB 2025-02-15 06:51:13,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:51:13,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47007.66 MB 2025-02-15 06:51:13,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52669.97 MB 2025-02-15 06:51:13,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:51:13,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49593.32 MB 2025-02-15 06:51:13,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:51:13,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:51:13,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 06:51:13,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39917.64 MB 2025-02-15 06:51:13,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44049.03 MB 2025-02-15 06:51:13,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:51:13,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46063.94 MB 2025-02-15 06:51:13,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52669.97 MB 2025-02-15 06:51:13,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 06:51:13,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49593.32 MB 2025-02-15 06:51:13,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:51:13,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:51:13,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:51:13,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45582.58 MB 2025-02-15 06:51:13,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46349.58 MB 2025-02-15 06:51:13,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:51:13,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52669.97 MB 2025-02-15 06:51:13,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 06:51:13,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 06:51:13,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47057.37 MB 2025-02-15 06:51:13,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:51:13,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:51:13,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:51:13,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46762.47 MB 2025-02-15 06:51:13,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46990.57 MB 2025-02-15 06:51:13,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-15 06:51:13,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 06:51:13,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 06:51:13,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:51:13,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47237.03 MB 2025-02-15 06:51:13,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:51:13,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:51:13,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.28 seconds 2025-02-15 06:51:13,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33049.66 MB 2025-02-15 06:51:13,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47191.42 MB 2025-02-15 06:51:13,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14141.76 MB 2025-02-15 06:51:13,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52518.98 MB 2025-02-15 06:51:13,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 06:51:13,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 564.13 MB 2025-02-15 06:51:13,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47237.03 MB 2025-02-15 06:51:13,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:51:13,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:51:13,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:51:13,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47191.42 MB 2025-02-15 06:51:13,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38045.81 MB 2025-02-15 06:51:13,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9145.61 MB 2025-02-15 06:51:13,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 06:51:13,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53083.11 MB 2025-02-15 06:51:13,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:51:13,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49696.64 MB 2025-02-15 06:51:13,552 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 06:51:13,553 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 06:51:13,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:51:13,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:51:13,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:51:13,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:51:13,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38045.81 MB 2025-02-15 06:51:13,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46463.55 MB 2025-02-15 06:51:13,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-15 06:51:13,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53083.11 MB 2025-02-15 06:51:13,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61450.75 MB 2025-02-15 06:51:13,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 06:51:13,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46463.55 MB 2025-02-15 06:51:13,720 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 06:51:13,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:51:13,722 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:51:13,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:51:13,723 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:51:13,728 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:51:13,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:51:13,729 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:51:13,729 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 06:52:12,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:12,099 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:52:12,104 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:52:12,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:12,108 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1410, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:52:12,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:12,109 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1410, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:52:33,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:52:33,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:52:33,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.83 seconds 2025-02-15 06:52:33,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:33,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35708.01 MB 2025-02-15 06:52:33,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40697.93 MB 2025-02-15 06:52:33,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4989.91 MB 2025-02-15 06:52:33,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69818.38 MB 2025-02-15 06:52:33,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53099.89 MB 2025-02-15 06:52:33,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16718.50 MB 2025-02-15 06:52:33,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49709.23 MB 2025-02-15 06:52:34,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:52:34,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:52:34,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:52:34,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:34,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40697.93 MB 2025-02-15 06:52:34,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36022.20 MB 2025-02-15 06:52:34,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4675.72 MB 2025-02-15 06:52:34,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53099.89 MB 2025-02-15 06:52:34,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62197.33 MB 2025-02-15 06:52:34,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9097.45 MB 2025-02-15 06:52:34,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54377.91 MB 2025-02-15 06:52:35,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:52:35,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:52:35,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 06:52:35,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:35,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36022.20 MB 2025-02-15 06:52:35,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36553.04 MB 2025-02-15 06:52:35,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:52:35,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62197.33 MB 2025-02-15 06:52:35,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48108.67 MB 2025-02-15 06:52:35,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14088.67 MB 2025-02-15 06:52:35,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40531.59 MB 2025-02-15 06:52:35,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:52:35,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:52:35,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:52:35,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:35,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36553.04 MB 2025-02-15 06:52:35,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38442.58 MB 2025-02-15 06:52:35,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:52:35,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48108.67 MB 2025-02-15 06:52:35,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48108.67 MB 2025-02-15 06:52:35,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:52:35,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39860.01 MB 2025-02-15 06:52:36,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:52:36,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:52:36,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:52:36,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38442.58 MB 2025-02-15 06:52:36,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40684.43 MB 2025-02-15 06:52:36,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:52:36,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48108.67 MB 2025-02-15 06:52:36,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49996.10 MB 2025-02-15 06:52:36,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:52:36,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46228.72 MB 2025-02-15 06:52:36,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:52:36,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:52:36,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:52:36,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36553.04 MB 2025-02-15 06:52:36,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40684.43 MB 2025-02-15 06:52:36,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:52:36,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48108.67 MB 2025-02-15 06:52:36,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49996.10 MB 2025-02-15 06:52:36,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:52:36,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46228.72 MB 2025-02-15 06:52:36,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:52:36,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:52:36,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:52:36,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42217.98 MB 2025-02-15 06:52:36,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42984.98 MB 2025-02-15 06:52:36,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:52:36,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49996.10 MB 2025-02-15 06:52:36,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50407.15 MB 2025-02-15 06:52:36,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 06:52:36,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43692.77 MB 2025-02-15 06:52:36,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:52:36,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:52:36,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:52:36,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43397.87 MB 2025-02-15 06:52:36,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43626.09 MB 2025-02-15 06:52:36,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 06:52:36,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50407.15 MB 2025-02-15 06:52:36,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50407.15 MB 2025-02-15 06:52:36,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:52:36,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43865.80 MB 2025-02-15 06:52:36,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:52:36,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:52:36,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.26 seconds 2025-02-15 06:52:36,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30795.46 MB 2025-02-15 06:52:36,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43826.77 MB 2025-02-15 06:52:36,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13031.31 MB 2025-02-15 06:52:36,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69818.38 MB 2025-02-15 06:52:36,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50407.15 MB 2025-02-15 06:52:36,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19411.24 MB 2025-02-15 06:52:36,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43865.80 MB 2025-02-15 06:52:36,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:52:36,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:52:36,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:52:36,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43826.77 MB 2025-02-15 06:52:36,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35793.39 MB 2025-02-15 06:52:36,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8033.38 MB 2025-02-15 06:52:36,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50407.15 MB 2025-02-15 06:52:36,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50407.15 MB 2025-02-15 06:52:36,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:52:36,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46333.69 MB 2025-02-15 06:52:36,657 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 06:52:36,657 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:52:36,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:52:36,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:52:36,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:52:36,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:52:36,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35793.39 MB 2025-02-15 06:52:36,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44215.71 MB 2025-02-15 06:52:36,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 06:52:36,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50407.15 MB 2025-02-15 06:52:36,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58781.07 MB 2025-02-15 06:52:36,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 06:52:36,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44215.71 MB 2025-02-15 06:52:36,824 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 06:52:36,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:36,826 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:52:36,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:36,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:52:36,832 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:52:36,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:52:36,834 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:52:36,834 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:53:58,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:53:58,556 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:53:58,561 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:53:58,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:53:58,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1351, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:53:58,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:53:58,566 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1351, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:54:19,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:54:19,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:54:19,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.84 seconds 2025-02-15 06:54:19,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:19,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35296.89 MB 2025-02-15 06:54:19,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40078.40 MB 2025-02-15 06:54:19,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4781.51 MB 2025-02-15 06:54:19,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71340.92 MB 2025-02-15 06:54:19,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52890.17 MB 2025-02-15 06:54:19,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18450.74 MB 2025-02-15 06:54:19,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49071.62 MB 2025-02-15 06:54:19,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:54:19,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:54:19,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 06:54:19,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:19,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40078.40 MB 2025-02-15 06:54:19,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35715.48 MB 2025-02-15 06:54:19,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4362.92 MB 2025-02-15 06:54:19,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52890.17 MB 2025-02-15 06:54:19,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62052.63 MB 2025-02-15 06:54:19,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9162.46 MB 2025-02-15 06:54:19,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53853.04 MB 2025-02-15 06:54:21,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:54:21,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:54:21,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:54:21,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35715.48 MB 2025-02-15 06:54:21,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36246.32 MB 2025-02-15 06:54:21,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:54:21,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62052.63 MB 2025-02-15 06:54:21,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43924.85 MB 2025-02-15 06:54:21,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18127.78 MB 2025-02-15 06:54:21,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40224.87 MB 2025-02-15 06:54:21,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:54:21,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:54:21,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:54:21,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36246.32 MB 2025-02-15 06:54:21,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38135.86 MB 2025-02-15 06:54:21,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:54:21,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:54:21,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43924.85 MB 2025-02-15 06:54:21,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:54:21,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39553.28 MB 2025-02-15 06:54:21,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:54:21,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:54:21,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:54:21,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38135.86 MB 2025-02-15 06:54:21,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40377.71 MB 2025-02-15 06:54:21,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:54:21,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:54:21,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49115.30 MB 2025-02-15 06:54:21,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 06:54:21,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45921.99 MB 2025-02-15 06:54:21,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:54:21,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:54:21,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:54:21,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36246.32 MB 2025-02-15 06:54:21,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40377.71 MB 2025-02-15 06:54:21,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:54:21,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43924.85 MB 2025-02-15 06:54:21,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49115.30 MB 2025-02-15 06:54:21,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 06:54:21,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45921.99 MB 2025-02-15 06:54:21,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:54:21,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:54:21,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:54:21,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41911.25 MB 2025-02-15 06:54:21,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42678.26 MB 2025-02-15 06:54:21,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:54:21,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49115.30 MB 2025-02-15 06:54:21,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49526.34 MB 2025-02-15 06:54:21,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 06:54:21,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43386.04 MB 2025-02-15 06:54:21,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:54:21,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:54:21,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:54:21,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43091.14 MB 2025-02-15 06:54:21,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43318.88 MB 2025-02-15 06:54:21,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.73 MB 2025-02-15 06:54:21,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49526.34 MB 2025-02-15 06:54:21,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49526.34 MB 2025-02-15 06:54:21,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:54:21,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43542.95 MB 2025-02-15 06:54:21,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:54:21,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:54:21,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.25 seconds 2025-02-15 06:54:21,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:21,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30589.90 MB 2025-02-15 06:54:21,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43518.97 MB 2025-02-15 06:54:21,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12929.07 MB 2025-02-15 06:54:21,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71340.92 MB 2025-02-15 06:54:21,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49526.34 MB 2025-02-15 06:54:21,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21814.58 MB 2025-02-15 06:54:21,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43542.95 MB 2025-02-15 06:54:22,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:54:22,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:54:22,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:54:22,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:22,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43518.97 MB 2025-02-15 06:54:22,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35578.68 MB 2025-02-15 06:54:22,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7940.29 MB 2025-02-15 06:54:22,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49526.34 MB 2025-02-15 06:54:22,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49526.34 MB 2025-02-15 06:54:22,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:54:22,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46018.35 MB 2025-02-15 06:54:22,107 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 06:54:22,108 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:54:22,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:54:22,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:54:22,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:54:22,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:54:22,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35578.68 MB 2025-02-15 06:54:22,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43976.08 MB 2025-02-15 06:54:22,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 06:54:22,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49526.34 MB 2025-02-15 06:54:22,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57877.20 MB 2025-02-15 06:54:22,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 06:54:22,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43976.08 MB 2025-02-15 06:54:22,279 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 06:54:22,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:54:22,280 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:54:22,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:54:22,281 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:54:22,286 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:54:22,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:54:22,287 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:54:22,287 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:56:04,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:04,852 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:56:04,857 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:56:04,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:04,860 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1668, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:56:04,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:04,861 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1668, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:56:30,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:56:30,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:56:30,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.67 seconds 2025-02-15 06:56:30,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:30,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37505.80 MB 2025-02-15 06:56:30,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43409.28 MB 2025-02-15 06:56:30,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5903.48 MB 2025-02-15 06:56:30,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66228.06 MB 2025-02-15 06:56:30,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53986.98 MB 2025-02-15 06:56:30,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12241.08 MB 2025-02-15 06:56:30,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52412.99 MB 2025-02-15 06:56:30,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:56:30,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:56:30,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:56:30,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:30,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43409.28 MB 2025-02-15 06:56:30,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37363.46 MB 2025-02-15 06:56:30,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6045.82 MB 2025-02-15 06:56:30,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53986.98 MB 2025-02-15 06:56:30,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66819.46 MB 2025-02-15 06:56:30,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12832.47 MB 2025-02-15 06:56:30,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60574.16 MB 2025-02-15 06:56:32,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:56:32,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:56:32,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 06:56:32,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37363.46 MB 2025-02-15 06:56:32,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37894.31 MB 2025-02-15 06:56:32,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:56:32,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66819.46 MB 2025-02-15 06:56:32,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49499.08 MB 2025-02-15 06:56:32,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17320.38 MB 2025-02-15 06:56:32,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41872.85 MB 2025-02-15 06:56:32,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:56:32,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:56:32,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:56:32,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37894.31 MB 2025-02-15 06:56:32,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39783.84 MB 2025-02-15 06:56:32,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:56:32,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49499.08 MB 2025-02-15 06:56:32,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49499.08 MB 2025-02-15 06:56:32,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:56:32,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41201.27 MB 2025-02-15 06:56:32,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:56:32,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:56:32,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:56:32,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39783.84 MB 2025-02-15 06:56:32,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42025.70 MB 2025-02-15 06:56:32,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:56:32,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49499.08 MB 2025-02-15 06:56:32,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52330.23 MB 2025-02-15 06:56:32,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:56:32,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47569.98 MB 2025-02-15 06:56:32,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:56:32,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:56:32,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:56:32,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37894.31 MB 2025-02-15 06:56:32,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42025.70 MB 2025-02-15 06:56:32,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:56:32,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49499.08 MB 2025-02-15 06:56:32,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52330.23 MB 2025-02-15 06:56:32,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 06:56:32,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47569.98 MB 2025-02-15 06:56:32,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:56:32,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:56:32,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:56:32,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43559.24 MB 2025-02-15 06:56:32,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44326.24 MB 2025-02-15 06:56:32,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:56:32,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52330.23 MB 2025-02-15 06:56:32,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52741.28 MB 2025-02-15 06:56:32,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 06:56:32,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45034.03 MB 2025-02-15 06:56:32,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:56:32,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:56:32,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:56:32,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44739.13 MB 2025-02-15 06:56:32,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44968.31 MB 2025-02-15 06:56:32,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.18 MB 2025-02-15 06:56:32,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52741.28 MB 2025-02-15 06:56:32,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52741.28 MB 2025-02-15 06:56:32,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:56:32,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45179.38 MB 2025-02-15 06:56:32,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:56:32,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:56:32,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.11 seconds 2025-02-15 06:56:32,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:32,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31694.35 MB 2025-02-15 06:56:32,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45168.94 MB 2025-02-15 06:56:32,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13474.58 MB 2025-02-15 06:56:32,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66228.06 MB 2025-02-15 06:56:32,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52741.28 MB 2025-02-15 06:56:32,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13486.78 MB 2025-02-15 06:56:32,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45179.38 MB 2025-02-15 06:56:33,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:56:33,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:56:33,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:56:33,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:33,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45168.94 MB 2025-02-15 06:56:33,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36691.57 MB 2025-02-15 06:56:33,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8477.37 MB 2025-02-15 06:56:33,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52741.28 MB 2025-02-15 06:56:33,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52741.28 MB 2025-02-15 06:56:33,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:56:33,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47675.30 MB 2025-02-15 06:56:33,261 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 06:56:33,261 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:56:33,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:56:33,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:56:33,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:56:33,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:56:33,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36691.57 MB 2025-02-15 06:56:33,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45112.34 MB 2025-02-15 06:56:33,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 06:56:33,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52741.28 MB 2025-02-15 06:56:33,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61113.11 MB 2025-02-15 06:56:33,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 06:56:33,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45112.34 MB 2025-02-15 06:56:33,425 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 06:56:33,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:33,427 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:56:33,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:33,428 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:56:33,433 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:56:33,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:33,434 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:56:33,434 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:56:40,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:40,854 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:56:40,860 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:56:40,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:40,863 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1991, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:56:40,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:56:40,864 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1991, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:57:11,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:57:11,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:57:11,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.02 seconds 2025-02-15 06:57:11,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:11,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39756.51 MB 2025-02-15 06:57:11,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46802.95 MB 2025-02-15 06:57:11,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7046.43 MB 2025-02-15 06:57:11,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69484.94 MB 2025-02-15 06:57:11,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55132.03 MB 2025-02-15 06:57:11,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14352.91 MB 2025-02-15 06:57:11,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55796.17 MB 2025-02-15 06:57:12,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:57:12,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:57:12,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:57:12,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:12,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46802.95 MB 2025-02-15 06:57:12,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39042.64 MB 2025-02-15 06:57:12,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7760.31 MB 2025-02-15 06:57:12,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55132.03 MB 2025-02-15 06:57:12,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69732.40 MB 2025-02-15 06:57:12,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14600.37 MB 2025-02-15 06:57:12,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66630.90 MB 2025-02-15 06:57:14,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:57:14,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:57:14,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 06:57:14,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39042.64 MB 2025-02-15 06:57:14,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39573.48 MB 2025-02-15 06:57:14,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 06:57:14,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69732.40 MB 2025-02-15 06:57:14,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45325.75 MB 2025-02-15 06:57:14,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24406.65 MB 2025-02-15 06:57:14,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43552.03 MB 2025-02-15 06:57:14,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:57:14,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:57:14,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:57:14,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39573.48 MB 2025-02-15 06:57:14,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41463.02 MB 2025-02-15 06:57:14,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 06:57:14,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45325.75 MB 2025-02-15 06:57:14,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 06:57:14,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 06:57:14,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42880.44 MB 2025-02-15 06:57:14,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:57:14,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:57:14,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 06:57:14,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41463.02 MB 2025-02-15 06:57:14,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43704.87 MB 2025-02-15 06:57:14,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 06:57:14,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 06:57:14,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52875.49 MB 2025-02-15 06:57:14,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 06:57:14,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49249.15 MB 2025-02-15 06:57:14,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:57:14,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:57:14,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 06:57:14,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39573.48 MB 2025-02-15 06:57:14,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43704.87 MB 2025-02-15 06:57:14,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 06:57:14,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45325.75 MB 2025-02-15 06:57:14,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52875.49 MB 2025-02-15 06:57:14,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 06:57:14,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49249.15 MB 2025-02-15 06:57:14,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:57:14,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:57:14,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 06:57:14,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45238.41 MB 2025-02-15 06:57:14,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46005.42 MB 2025-02-15 06:57:14,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 06:57:14,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52875.49 MB 2025-02-15 06:57:14,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53284.44 MB 2025-02-15 06:57:14,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 06:57:14,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46713.20 MB 2025-02-15 06:57:14,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:57:14,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:57:14,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:57:14,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46418.31 MB 2025-02-15 06:57:14,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46646.82 MB 2025-02-15 06:57:14,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-15 06:57:14,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53284.44 MB 2025-02-15 06:57:14,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53284.44 MB 2025-02-15 06:57:14,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:14,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46872.18 MB 2025-02-15 06:57:14,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:57:14,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:57:14,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.60 seconds 2025-02-15 06:57:14,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32819.71 MB 2025-02-15 06:57:14,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46847.80 MB 2025-02-15 06:57:14,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14028.09 MB 2025-02-15 06:57:14,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69484.94 MB 2025-02-15 06:57:14,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53284.44 MB 2025-02-15 06:57:14,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16200.50 MB 2025-02-15 06:57:14,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46872.18 MB 2025-02-15 06:57:14,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:57:14,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:57:14,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:57:14,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46847.80 MB 2025-02-15 06:57:14,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37822.04 MB 2025-02-15 06:57:14,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9025.76 MB 2025-02-15 06:57:14,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53284.44 MB 2025-02-15 06:57:14,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53284.44 MB 2025-02-15 06:57:14,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:14,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49358.24 MB 2025-02-15 06:57:14,755 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 06:57:14,755 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:57:14,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:57:14,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:57:14,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:57:14,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:14,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37822.04 MB 2025-02-15 06:57:14,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46256.89 MB 2025-02-15 06:57:14,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 06:57:14,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53284.44 MB 2025-02-15 06:57:14,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61670.95 MB 2025-02-15 06:57:14,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 06:57:14,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46256.89 MB 2025-02-15 06:57:14,918 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 06:57:14,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:14,920 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:57:14,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:14,921 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:57:14,925 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:57:14,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:14,926 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:57:14,926 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:57:27,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:27,064 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:57:27,069 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:57:27,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:27,072 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:57:27,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:27,073 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:57:28,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:57:28,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:57:28,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.37 seconds 2025-02-15 06:57:28,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26482.17 MB 2025-02-15 06:57:28,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26786.52 MB 2025-02-15 06:57:28,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-15 06:57:28,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74249.67 MB 2025-02-15 06:57:28,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38692.45 MB 2025-02-15 06:57:28,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35727.05 MB 2025-02-15 06:57:28,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:57:28,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:57:28,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:57:28,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26786.52 MB 2025-02-15 06:57:28,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26933.98 MB 2025-02-15 06:57:28,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-15 06:57:28,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:28,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:28,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27390.56 MB 2025-02-15 06:57:28,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:57:28,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:57:28,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-15 06:57:28,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26933.98 MB 2025-02-15 06:57:28,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.11 MB 2025-02-15 06:57:28,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.13 MB 2025-02-15 06:57:28,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:28,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:28,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31018.69 MB 2025-02-15 06:57:28,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:57:28,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:57:28,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:57:28,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.04 MB 2025-02-15 06:57:28,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27454.19 MB 2025-02-15 06:57:28,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-15 06:57:28,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:28,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:28,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27758.95 MB 2025-02-15 06:57:28,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:57:28,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:57:28,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 06:57:28,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27454.19 MB 2025-02-15 06:57:28,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27947.51 MB 2025-02-15 06:57:28,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-15 06:57:28,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:28,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:28,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29128.21 MB 2025-02-15 06:57:28,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:57:28,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:57:28,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 06:57:28,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:28,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.04 MB 2025-02-15 06:57:28,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27947.51 MB 2025-02-15 06:57:28,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-15 06:57:28,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:28,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 06:57:28,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:28,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29128.21 MB 2025-02-15 06:57:29,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:57:29,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:57:29,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 06:57:29,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:29,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28424.49 MB 2025-02-15 06:57:29,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28631.66 MB 2025-02-15 06:57:29,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.18 MB 2025-02-15 06:57:29,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 06:57:29,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35683.04 MB 2025-02-15 06:57:29,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 125.83 MB 2025-02-15 06:57:29,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28783.84 MB 2025-02-15 06:57:29,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:57:29,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:57:29,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 06:57:29,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:29,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28762.71 MB 2025-02-15 06:57:29,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28968.03 MB 2025-02-15 06:57:29,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.31 MB 2025-02-15 06:57:29,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35683.04 MB 2025-02-15 06:57:29,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35683.04 MB 2025-02-15 06:57:29,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:29,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28968.03 MB 2025-02-15 06:57:29,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:57:29,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:57:29,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 06:57:29,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:29,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26182.54 MB 2025-02-15 06:57:29,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29151.51 MB 2025-02-15 06:57:29,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2968.97 MB 2025-02-15 06:57:29,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74249.67 MB 2025-02-15 06:57:29,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35683.04 MB 2025-02-15 06:57:29,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38566.63 MB 2025-02-15 06:57:29,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29151.51 MB 2025-02-15 06:57:29,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:57:29,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:57:29,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 06:57:29,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:29,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26681.69 MB 2025-02-15 06:57:29,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29432.14 MB 2025-02-15 06:57:29,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-15 06:57:29,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35683.04 MB 2025-02-15 06:57:29,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35683.04 MB 2025-02-15 06:57:29,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:29,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29707.15 MB 2025-02-15 06:57:29,283 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-15 06:57:29,283 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:57:29,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:57:29,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:57:29,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:57:29,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:29,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29432.14 MB 2025-02-15 06:57:29,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37132.96 MB 2025-02-15 06:57:29,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-15 06:57:29,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35683.04 MB 2025-02-15 06:57:29,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43339.74 MB 2025-02-15 06:57:29,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7656.70 MB 2025-02-15 06:57:29,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37132.96 MB 2025-02-15 06:57:29,433 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-15 06:57:29,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:29,434 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:57:29,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:29,435 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:57:29,440 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:57:29,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:29,441 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:57:29,441 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 06:57:51,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:51,280 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:57:51,288 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:57:51,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:51,294 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 235, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:57:51,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:51,296 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 235, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:57:55,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:57:55,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:57:55,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.73 seconds 2025-02-15 06:57:55,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:55,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27520.43 MB 2025-02-15 06:57:55,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28352.08 MB 2025-02-15 06:57:55,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.65 MB 2025-02-15 06:57:55,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54823.75 MB 2025-02-15 06:57:55,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:57:55,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19268.63 MB 2025-02-15 06:57:55,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37218.29 MB 2025-02-15 06:57:55,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:57:55,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:57:55,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:57:55,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:55,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28352.08 MB 2025-02-15 06:57:55,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28754.95 MB 2025-02-15 06:57:55,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.87 MB 2025-02-15 06:57:55,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:57:55,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:57:55,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:55,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31652.90 MB 2025-02-15 06:57:56,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:57:56,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:57:56,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.14 seconds 2025-02-15 06:57:56,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28754.95 MB 2025-02-15 06:57:56,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29066.82 MB 2025-02-15 06:57:56,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.87 MB 2025-02-15 06:57:56,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:57:56,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:57:56,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:56,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33009.53 MB 2025-02-15 06:57:56,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:57:56,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:57:56,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:57:56,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29066.82 MB 2025-02-15 06:57:56,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30176.65 MB 2025-02-15 06:57:56,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.83 MB 2025-02-15 06:57:56,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:57:56,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:57:56,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:56,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31009.39 MB 2025-02-15 06:57:56,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:57:56,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:57:56,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 06:57:56,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30176.65 MB 2025-02-15 06:57:56,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31493.76 MB 2025-02-15 06:57:56,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1317.12 MB 2025-02-15 06:57:56,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:57:56,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-15 06:57:56,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1667.24 MB 2025-02-15 06:57:56,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34751.00 MB 2025-02-15 06:57:56,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:57:56,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:57:56,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 06:57:56,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29066.82 MB 2025-02-15 06:57:56,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31493.76 MB 2025-02-15 06:57:56,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2426.95 MB 2025-02-15 06:57:56,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:57:56,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-15 06:57:56,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1667.24 MB 2025-02-15 06:57:56,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34751.00 MB 2025-02-15 06:57:56,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:57:56,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:57:56,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 06:57:56,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32394.72 MB 2025-02-15 06:57:56,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32845.34 MB 2025-02-15 06:57:56,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 450.61 MB 2025-02-15 06:57:56,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37222.35 MB 2025-02-15 06:57:56,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 06:57:56,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 241.17 MB 2025-02-15 06:57:56,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33261.16 MB 2025-02-15 06:57:56,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:57:56,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:57:56,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:57:56,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33087.91 MB 2025-02-15 06:57:56,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33295.84 MB 2025-02-15 06:57:56,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.93 MB 2025-02-15 06:57:56,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 06:57:56,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 06:57:56,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:57:56,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33362.36 MB 2025-02-15 06:57:56,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:57:56,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:57:56,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.16 seconds 2025-02-15 06:57:56,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26701.67 MB 2025-02-15 06:57:56,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33496.92 MB 2025-02-15 06:57:56,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6795.25 MB 2025-02-15 06:57:56,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54823.75 MB 2025-02-15 06:57:56,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 06:57:56,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17360.22 MB 2025-02-15 06:57:56,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.92 MB 2025-02-15 06:57:56,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:57:56,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:57:56,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:57:56,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33496.92 MB 2025-02-15 06:57:56,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36510.95 MB 2025-02-15 06:57:56,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 06:57:56,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 06:57:56,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37731.96 MB 2025-02-15 06:57:56,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-15 06:57:56,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36812.58 MB 2025-02-15 06:57:56,744 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:57:56,744 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:57:56,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:57:56,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:57:56,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:57:56,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:57:56,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30926.83 MB 2025-02-15 06:57:56,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39365.86 MB 2025-02-15 06:57:56,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:57:56,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37731.96 MB 2025-02-15 06:57:56,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46122.66 MB 2025-02-15 06:57:56,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:57:56,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39365.86 MB 2025-02-15 06:57:56,918 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:57:56,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:56,919 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:57:56,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:56,920 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:57:56,925 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:57:56,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:57:56,926 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:57:56,927 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 06:59:15,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:15,314 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:59:15,319 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:59:15,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:15,323 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 354, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:59:15,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:15,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 354, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 06:59:20,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 06:59:20,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 06:59:20,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.41 seconds 2025-02-15 06:59:20,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:20,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28349.64 MB 2025-02-15 06:59:20,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29602.42 MB 2025-02-15 06:59:20,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1252.79 MB 2025-02-15 06:59:20,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58707.67 MB 2025-02-15 06:59:20,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 06:59:20,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23152.56 MB 2025-02-15 06:59:20,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38500.49 MB 2025-02-15 06:59:20,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 06:59:20,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 06:59:20,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:59:20,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:20,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29602.42 MB 2025-02-15 06:59:20,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30204.11 MB 2025-02-15 06:59:20,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.69 MB 2025-02-15 06:59:20,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 06:59:20,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38036.05 MB 2025-02-15 06:59:20,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-15 06:59:20,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.48 MB 2025-02-15 06:59:22,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 06:59:22,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 06:59:22,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.67 seconds 2025-02-15 06:59:22,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30204.11 MB 2025-02-15 06:59:22,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30672.58 MB 2025-02-15 06:59:22,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 468.47 MB 2025-02-15 06:59:22,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38036.05 MB 2025-02-15 06:59:22,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38036.05 MB 2025-02-15 06:59:22,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:59:22,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34628.56 MB 2025-02-15 06:59:22,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 06:59:22,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 06:59:22,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:59:22,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30672.58 MB 2025-02-15 06:59:22,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32339.80 MB 2025-02-15 06:59:22,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1667.22 MB 2025-02-15 06:59:22,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38036.05 MB 2025-02-15 06:59:22,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38036.05 MB 2025-02-15 06:59:22,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:59:22,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33590.68 MB 2025-02-15 06:59:22,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 06:59:22,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 06:59:22,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 06:59:22,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32339.80 MB 2025-02-15 06:59:22,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34318.69 MB 2025-02-15 06:59:22,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.89 MB 2025-02-15 06:59:22,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38036.05 MB 2025-02-15 06:59:22,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41792.05 MB 2025-02-15 06:59:22,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3756.00 MB 2025-02-15 06:59:22,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39212.43 MB 2025-02-15 06:59:22,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 06:59:22,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 06:59:22,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 06:59:22,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30672.58 MB 2025-02-15 06:59:22,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34318.69 MB 2025-02-15 06:59:22,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3646.11 MB 2025-02-15 06:59:22,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38036.05 MB 2025-02-15 06:59:22,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41792.05 MB 2025-02-15 06:59:22,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3756.00 MB 2025-02-15 06:59:22,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39212.43 MB 2025-02-15 06:59:22,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 06:59:22,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 06:59:22,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 06:59:22,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35672.04 MB 2025-02-15 06:59:22,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36349.84 MB 2025-02-15 06:59:22,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 677.80 MB 2025-02-15 06:59:22,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41792.05 MB 2025-02-15 06:59:22,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42154.85 MB 2025-02-15 06:59:22,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 362.81 MB 2025-02-15 06:59:22,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36974.46 MB 2025-02-15 06:59:22,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 06:59:22,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 06:59:22,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 06:59:22,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36714.21 MB 2025-02-15 06:59:22,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36945.24 MB 2025-02-15 06:59:22,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.03 MB 2025-02-15 06:59:22,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42154.85 MB 2025-02-15 06:59:22,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42154.85 MB 2025-02-15 06:59:22,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:59:22,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37107.91 MB 2025-02-15 06:59:22,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 06:59:22,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 06:59:22,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.49 seconds 2025-02-15 06:59:22,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:22,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.27 MB 2025-02-15 06:59:22,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37146.32 MB 2025-02-15 06:59:22,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10030.04 MB 2025-02-15 06:59:22,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58707.67 MB 2025-02-15 06:59:22,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42156.95 MB 2025-02-15 06:59:22,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16550.72 MB 2025-02-15 06:59:22,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37146.32 MB 2025-02-15 06:59:23,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 06:59:23,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 06:59:23,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 06:59:23,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:23,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37146.32 MB 2025-02-15 06:59:23,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31899.77 MB 2025-02-15 06:59:23,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5246.54 MB 2025-02-15 06:59:23,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42156.95 MB 2025-02-15 06:59:23,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42156.95 MB 2025-02-15 06:59:23,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 06:59:23,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40361.25 MB 2025-02-15 06:59:23,100 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 06:59:23,100 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 06:59:23,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 06:59:23,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 06:59:23,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 06:59:23,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 06:59:23,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31899.77 MB 2025-02-15 06:59:23,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40338.80 MB 2025-02-15 06:59:23,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 06:59:23,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42156.95 MB 2025-02-15 06:59:23,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50547.65 MB 2025-02-15 06:59:23,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 06:59:23,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40338.80 MB 2025-02-15 06:59:23,274 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 06:59:23,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:23,275 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 06:59:23,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:23,276 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 06:59:23,281 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 06:59:23,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:23,282 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 06:59:23,282 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 06:59:36,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:36,355 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 06:59:36,363 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 06:59:36,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:36,370 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1810, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 06:59:36,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 06:59:36,372 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1810, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:00:04,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:00:04,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:00:04,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.09 seconds 2025-02-15 07:00:04,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:04,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38495.28 MB 2025-02-15 07:00:04,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44900.77 MB 2025-02-15 07:00:04,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6405.49 MB 2025-02-15 07:00:04,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63132.66 MB 2025-02-15 07:00:04,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54544.83 MB 2025-02-15 07:00:04,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8587.84 MB 2025-02-15 07:00:04,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53855.45 MB 2025-02-15 07:00:04,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:00:04,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:00:04,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 07:00:04,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:04,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44900.77 MB 2025-02-15 07:00:04,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38101.68 MB 2025-02-15 07:00:04,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6799.09 MB 2025-02-15 07:00:04,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54544.83 MB 2025-02-15 07:00:04,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68295.85 MB 2025-02-15 07:00:04,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13751.03 MB 2025-02-15 07:00:04,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63450.81 MB 2025-02-15 07:00:06,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:00:06,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:00:06,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:00:06,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38101.68 MB 2025-02-15 07:00:06,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38632.52 MB 2025-02-15 07:00:06,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:00:06,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68295.85 MB 2025-02-15 07:00:06,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45359.30 MB 2025-02-15 07:00:06,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22936.55 MB 2025-02-15 07:00:06,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42611.07 MB 2025-02-15 07:00:06,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:00:06,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:00:06,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:00:06,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38632.52 MB 2025-02-15 07:00:06,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40522.05 MB 2025-02-15 07:00:06,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:00:06,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45359.30 MB 2025-02-15 07:00:06,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46303.02 MB 2025-02-15 07:00:06,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:00:06,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41939.48 MB 2025-02-15 07:00:06,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:00:06,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:00:06,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:00:06,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40522.05 MB 2025-02-15 07:00:06,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42763.91 MB 2025-02-15 07:00:06,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:00:06,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46303.02 MB 2025-02-15 07:00:06,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51965.33 MB 2025-02-15 07:00:06,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:00:06,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48308.19 MB 2025-02-15 07:00:06,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:00:06,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:00:06,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:00:06,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38632.52 MB 2025-02-15 07:00:06,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42763.91 MB 2025-02-15 07:00:06,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:00:06,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45359.30 MB 2025-02-15 07:00:06,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51965.33 MB 2025-02-15 07:00:06,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:00:06,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48308.19 MB 2025-02-15 07:00:06,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:00:06,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:00:06,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:00:06,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44297.45 MB 2025-02-15 07:00:06,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45064.45 MB 2025-02-15 07:00:06,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:00:06,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51965.33 MB 2025-02-15 07:00:06,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:00:06,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:00:06,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45772.24 MB 2025-02-15 07:00:06,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:00:06,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:00:06,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:00:06,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45477.83 MB 2025-02-15 07:00:06,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45706.45 MB 2025-02-15 07:00:06,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-15 07:00:06,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:00:06,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:00:06,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:06,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45933.64 MB 2025-02-15 07:00:06,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:00:06,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:00:06,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.58 seconds 2025-02-15 07:00:06,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:06,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32189.09 MB 2025-02-15 07:00:06,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45907.30 MB 2025-02-15 07:00:06,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13718.21 MB 2025-02-15 07:00:06,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63132.66 MB 2025-02-15 07:00:06,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:00:06,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10754.20 MB 2025-02-15 07:00:06,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45933.64 MB 2025-02-15 07:00:07,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:00:07,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:00:07,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:00:07,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:07,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45907.30 MB 2025-02-15 07:00:07,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37189.16 MB 2025-02-15 07:00:07,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8718.14 MB 2025-02-15 07:00:07,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:00:07,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:00:07,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:07,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48415.89 MB 2025-02-15 07:00:07,242 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 07:00:07,242 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:00:07,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:00:07,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:00:07,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:00:07,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:07,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37189.16 MB 2025-02-15 07:00:07,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45618.28 MB 2025-02-15 07:00:07,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 07:00:07,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:00:07,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60758.69 MB 2025-02-15 07:00:07,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 07:00:07,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45618.28 MB 2025-02-15 07:00:07,417 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 07:00:07,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:07,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:00:07,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:07,420 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:00:07,425 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:00:07,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:07,426 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:00:07,426 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:00:17,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:17,340 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:00:17,345 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:00:17,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:17,348 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 229, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:00:17,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:17,349 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 229, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:00:20,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:00:20,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:00:20,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.56 seconds 2025-02-15 07:00:20,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:20,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27478.62 MB 2025-02-15 07:00:20,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28289.04 MB 2025-02-15 07:00:20,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 810.42 MB 2025-02-15 07:00:20,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69138.91 MB 2025-02-15 07:00:20,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:20,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31685.87 MB 2025-02-15 07:00:20,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37176.48 MB 2025-02-15 07:00:20,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:00:20,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:00:20,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:00:20,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:20,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28289.04 MB 2025-02-15 07:00:20,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28534.13 MB 2025-02-15 07:00:20,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.10 MB 2025-02-15 07:00:20,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:20,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:20,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:20,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.61 MB 2025-02-15 07:00:21,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:00:21,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:00:21,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-15 07:00:21,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:21,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28534.13 MB 2025-02-15 07:00:21,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28810.17 MB 2025-02-15 07:00:21,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-15 07:00:21,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:21,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:21,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:21,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32788.97 MB 2025-02-15 07:00:21,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:00:21,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:00:21,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:00:21,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:21,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28810.17 MB 2025-02-15 07:00:21,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29792.49 MB 2025-02-15 07:00:21,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-15 07:00:21,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:21,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:21,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:21,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30529.56 MB 2025-02-15 07:00:22,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:00:22,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:00:22,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:00:22,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29792.49 MB 2025-02-15 07:00:22,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30958.28 MB 2025-02-15 07:00:22,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1165.80 MB 2025-02-15 07:00:22,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:22,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:22,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:22,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33841.28 MB 2025-02-15 07:00:22,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:00:22,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:00:22,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 07:00:22,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28810.17 MB 2025-02-15 07:00:22,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30958.28 MB 2025-02-15 07:00:22,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.11 MB 2025-02-15 07:00:22,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:22,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-15 07:00:22,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:22,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33841.28 MB 2025-02-15 07:00:22,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:00:22,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:00:22,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:00:22,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31755.73 MB 2025-02-15 07:00:22,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32154.57 MB 2025-02-15 07:00:22,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-15 07:00:22,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-15 07:00:22,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37660.66 MB 2025-02-15 07:00:22,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-15 07:00:22,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32524.69 MB 2025-02-15 07:00:22,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:00:22,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:00:22,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:00:22,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32369.28 MB 2025-02-15 07:00:22,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32597.51 MB 2025-02-15 07:00:22,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.24 MB 2025-02-15 07:00:22,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37660.66 MB 2025-02-15 07:00:22,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37660.66 MB 2025-02-15 07:00:22,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:22,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.10 MB 2025-02-15 07:00:22,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:00:22,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:00:22,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.80 seconds 2025-02-15 07:00:22,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26680.76 MB 2025-02-15 07:00:22,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32798.12 MB 2025-02-15 07:00:22,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6117.35 MB 2025-02-15 07:00:22,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69138.91 MB 2025-02-15 07:00:22,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37660.66 MB 2025-02-15 07:00:22,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31478.25 MB 2025-02-15 07:00:22,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32798.12 MB 2025-02-15 07:00:22,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:00:22,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:00:22,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:00:22,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27764.24 MB 2025-02-15 07:00:22,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30771.38 MB 2025-02-15 07:00:22,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.14 MB 2025-02-15 07:00:22,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37660.66 MB 2025-02-15 07:00:22,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37660.66 MB 2025-02-15 07:00:22,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:00:22,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31072.05 MB 2025-02-15 07:00:22,442 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 07:00:22,443 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:00:22,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:00:22,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:00:22,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:00:22,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:00:22,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30771.38 MB 2025-02-15 07:00:22,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39190.46 MB 2025-02-15 07:00:22,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 07:00:22,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37660.66 MB 2025-02-15 07:00:22,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46032.49 MB 2025-02-15 07:00:22,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 07:00:22,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39190.46 MB 2025-02-15 07:00:22,607 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 07:00:22,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:22,608 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:00:22,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:22,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:00:22,614 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:00:22,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:00:22,615 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:00:22,615 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:01:18,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:18,654 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:01:18,659 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:01:18,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:18,663 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:01:18,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:18,664 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:01:21,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:01:21,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:01:21,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.47 seconds 2025-02-15 07:01:21,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:21,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27004.78 MB 2025-02-15 07:01:21,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27574.55 MB 2025-02-15 07:01:21,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 07:01:21,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54404.32 MB 2025-02-15 07:01:21,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:21,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18849.20 MB 2025-02-15 07:01:21,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36476.15 MB 2025-02-15 07:01:21,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:01:21,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:01:21,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:01:21,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:21,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27574.55 MB 2025-02-15 07:01:21,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27850.60 MB 2025-02-15 07:01:21,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-15 07:01:21,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:21,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:21,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:21,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29836.03 MB 2025-02-15 07:01:21,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:01:21,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:01:21,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-15 07:01:21,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:21,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27850.60 MB 2025-02-15 07:01:21,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28064.27 MB 2025-02-15 07:01:21,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 07:01:21,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:21,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:21,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:21,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32020.25 MB 2025-02-15 07:01:21,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:01:21,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:01:21,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:01:21,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:21,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28064.20 MB 2025-02-15 07:01:21,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28824.56 MB 2025-02-15 07:01:21,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 07:01:21,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:21,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:21,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:21,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29395.08 MB 2025-02-15 07:01:22,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:01:22,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:01:22,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:01:22,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28824.56 MB 2025-02-15 07:01:22,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29726.94 MB 2025-02-15 07:01:22,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 07:01:22,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:22,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:22,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:22,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31958.48 MB 2025-02-15 07:01:22,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:01:22,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:01:22,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:01:22,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28064.20 MB 2025-02-15 07:01:22,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29726.94 MB 2025-02-15 07:01:22,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 07:01:22,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:22,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-15 07:01:22,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:22,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31958.48 MB 2025-02-15 07:01:22,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:01:22,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:01:22,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:01:22,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30344.19 MB 2025-02-15 07:01:22,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30652.91 MB 2025-02-15 07:01:22,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 07:01:22,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-15 07:01:22,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:01:22,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 07:01:22,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30945.91 MB 2025-02-15 07:01:22,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:01:22,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:01:22,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:01:22,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30819.11 MB 2025-02-15 07:01:22,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31046.32 MB 2025-02-15 07:01:22,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.21 MB 2025-02-15 07:01:22,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:01:22,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:01:22,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:22,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31063.49 MB 2025-02-15 07:01:22,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:01:22,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:01:22,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.43 seconds 2025-02-15 07:01:22,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26443.85 MB 2025-02-15 07:01:22,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31247.09 MB 2025-02-15 07:01:22,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4803.25 MB 2025-02-15 07:01:22,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54404.32 MB 2025-02-15 07:01:22,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:01:22,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18685.62 MB 2025-02-15 07:01:22,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31247.09 MB 2025-02-15 07:01:22,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:01:22,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:01:22,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:01:22,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31247.09 MB 2025-02-15 07:01:22,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30315.21 MB 2025-02-15 07:01:22,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -931.88 MB 2025-02-15 07:01:22,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:01:22,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:01:22,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:22,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32049.65 MB 2025-02-15 07:01:22,378 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 07:01:22,379 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-15 07:01:22,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:01:22,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:01:22,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:01:22,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:22,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30315.21 MB 2025-02-15 07:01:22,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38741.71 MB 2025-02-15 07:01:22,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 07:01:22,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:01:22,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44096.82 MB 2025-02-15 07:01:22,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-15 07:01:22,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38741.71 MB 2025-02-15 07:01:22,546 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 07:01:22,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:22,548 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:01:22,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:22,550 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:01:22,554 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:01:22,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:22,555 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:01:22,556 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-15 07:01:34,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:34,659 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:01:34,664 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:01:34,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:34,667 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1190, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:01:34,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:34,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1190, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:01:53,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:01:53,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:01:53,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.36 seconds 2025-02-15 07:01:53,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:53,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34175.02 MB 2025-02-15 07:01:53,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38386.36 MB 2025-02-15 07:01:53,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4211.34 MB 2025-02-15 07:01:53,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56662.95 MB 2025-02-15 07:01:53,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43956.31 MB 2025-02-15 07:01:53,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12706.64 MB 2025-02-15 07:01:53,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47271.08 MB 2025-02-15 07:01:53,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:01:53,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:01:53,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:01:53,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:53,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38386.36 MB 2025-02-15 07:01:53,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34878.49 MB 2025-02-15 07:01:53,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3507.87 MB 2025-02-15 07:01:53,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43956.31 MB 2025-02-15 07:01:53,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53703.87 MB 2025-02-15 07:01:53,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9747.56 MB 2025-02-15 07:01:53,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50941.82 MB 2025-02-15 07:01:55,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:01:55,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:01:55,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:01:55,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34878.49 MB 2025-02-15 07:01:55,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35409.33 MB 2025-02-15 07:01:55,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:01:55,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53703.87 MB 2025-02-15 07:01:55,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41158.71 MB 2025-02-15 07:01:55,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12545.16 MB 2025-02-15 07:01:55,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39387.88 MB 2025-02-15 07:01:55,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:01:55,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:01:55,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:01:55,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35409.33 MB 2025-02-15 07:01:55,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37298.87 MB 2025-02-15 07:01:55,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:01:55,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41158.71 MB 2025-02-15 07:01:55,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42102.42 MB 2025-02-15 07:01:55,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:01:55,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38716.30 MB 2025-02-15 07:01:55,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:01:55,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:01:55,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:01:55,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37298.87 MB 2025-02-15 07:01:55,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39540.72 MB 2025-02-15 07:01:55,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:01:55,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42102.42 MB 2025-02-15 07:01:55,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47764.73 MB 2025-02-15 07:01:55,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:01:55,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45085.01 MB 2025-02-15 07:01:55,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:01:55,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:01:55,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:01:55,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35409.33 MB 2025-02-15 07:01:55,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39540.72 MB 2025-02-15 07:01:55,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:01:55,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41158.71 MB 2025-02-15 07:01:55,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47764.73 MB 2025-02-15 07:01:55,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:01:55,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45085.01 MB 2025-02-15 07:01:55,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:01:55,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:01:55,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:01:55,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41074.27 MB 2025-02-15 07:01:55,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41841.27 MB 2025-02-15 07:01:55,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:01:55,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47764.73 MB 2025-02-15 07:01:55,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48177.87 MB 2025-02-15 07:01:55,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:01:55,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42549.06 MB 2025-02-15 07:01:55,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:01:55,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:01:55,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:01:55,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42254.16 MB 2025-02-15 07:01:55,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42482.13 MB 2025-02-15 07:01:55,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-15 07:01:55,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48177.87 MB 2025-02-15 07:01:55,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48177.87 MB 2025-02-15 07:01:55,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:55,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42717.94 MB 2025-02-15 07:01:55,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:01:55,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:01:55,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.78 seconds 2025-02-15 07:01:55,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30028.96 MB 2025-02-15 07:01:55,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42682.57 MB 2025-02-15 07:01:55,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12653.60 MB 2025-02-15 07:01:55,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56662.95 MB 2025-02-15 07:01:55,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48177.87 MB 2025-02-15 07:01:55,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8485.08 MB 2025-02-15 07:01:55,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42717.94 MB 2025-02-15 07:01:55,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:01:55,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:01:55,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:01:55,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42682.57 MB 2025-02-15 07:01:55,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35023.33 MB 2025-02-15 07:01:55,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7659.24 MB 2025-02-15 07:01:55,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48177.87 MB 2025-02-15 07:01:55,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48177.87 MB 2025-02-15 07:01:55,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:01:55,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45186.67 MB 2025-02-15 07:01:55,740 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 07:01:55,740 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:01:55,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:01:55,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:01:55,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:01:55,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:01:55,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35023.33 MB 2025-02-15 07:01:55,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43435.76 MB 2025-02-15 07:01:55,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-15 07:01:55,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48177.87 MB 2025-02-15 07:01:55,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56541.32 MB 2025-02-15 07:01:55,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 07:01:55,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43435.76 MB 2025-02-15 07:01:55,904 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 07:01:55,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:55,905 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:01:55,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:55,906 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:01:55,911 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:01:55,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:01:55,912 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:01:55,912 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:03:22,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:22,064 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:03:22,072 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:03:22,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:22,079 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:03:22,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:22,081 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:03:25,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:03:25,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:03:25,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.34 seconds 2025-02-15 07:03:25,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:25,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27353.19 MB 2025-02-15 07:03:25,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28099.91 MB 2025-02-15 07:03:25,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-15 07:03:25,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64904.76 MB 2025-02-15 07:03:25,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:25,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28401.73 MB 2025-02-15 07:03:25,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37051.05 MB 2025-02-15 07:03:25,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:03:25,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:03:25,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:03:25,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:25,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28099.91 MB 2025-02-15 07:03:25,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28384.44 MB 2025-02-15 07:03:25,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.53 MB 2025-02-15 07:03:25,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:25,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:25,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:25,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30909.18 MB 2025-02-15 07:03:26,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:03:26,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:03:26,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-15 07:03:26,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28384.44 MB 2025-02-15 07:03:26,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28649.86 MB 2025-02-15 07:03:26,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 265.42 MB 2025-02-15 07:03:26,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:26,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:26,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32639.02 MB 2025-02-15 07:03:26,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:03:26,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:03:26,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:03:26,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28649.79 MB 2025-02-15 07:03:26,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29594.33 MB 2025-02-15 07:03:26,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 944.54 MB 2025-02-15 07:03:26,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:26,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:26,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30303.05 MB 2025-02-15 07:03:26,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:03:26,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:03:26,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:03:26,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29594.33 MB 2025-02-15 07:03:26,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30715.29 MB 2025-02-15 07:03:26,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1120.96 MB 2025-02-15 07:03:26,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:26,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:26,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.40 MB 2025-02-15 07:03:26,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:03:26,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:03:26,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:03:26,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28649.79 MB 2025-02-15 07:03:26,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30715.29 MB 2025-02-15 07:03:26,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2065.50 MB 2025-02-15 07:03:26,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:26,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:03:26,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.40 MB 2025-02-15 07:03:26,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:03:26,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:03:26,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:03:26,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31482.06 MB 2025-02-15 07:03:26,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31865.56 MB 2025-02-15 07:03:26,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 383.50 MB 2025-02-15 07:03:26,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:03:26,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36702.26 MB 2025-02-15 07:03:26,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-15 07:03:26,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32220.35 MB 2025-02-15 07:03:26,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:03:26,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:03:26,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:03:26,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32072.01 MB 2025-02-15 07:03:26,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32300.86 MB 2025-02-15 07:03:26,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.85 MB 2025-02-15 07:03:26,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36702.26 MB 2025-02-15 07:03:26,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36702.26 MB 2025-02-15 07:03:26,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32352.43 MB 2025-02-15 07:03:26,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:03:26,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:03:26,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.55 seconds 2025-02-15 07:03:26,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26618.05 MB 2025-02-15 07:03:26,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32501.57 MB 2025-02-15 07:03:26,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5883.52 MB 2025-02-15 07:03:26,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64904.76 MB 2025-02-15 07:03:26,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36702.26 MB 2025-02-15 07:03:26,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28202.50 MB 2025-02-15 07:03:26,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32501.57 MB 2025-02-15 07:03:26,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:03:26,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:03:26,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 07:03:26,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27663.82 MB 2025-02-15 07:03:26,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30672.33 MB 2025-02-15 07:03:26,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-15 07:03:26,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36702.26 MB 2025-02-15 07:03:26,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36702.26 MB 2025-02-15 07:03:26,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:03:26,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30973.14 MB 2025-02-15 07:03:26,916 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 07:03:26,916 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:03:26,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:03:26,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:03:26,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:03:26,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:03:26,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30672.33 MB 2025-02-15 07:03:26,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39095.53 MB 2025-02-15 07:03:26,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 07:03:26,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36702.26 MB 2025-02-15 07:03:26,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45078.28 MB 2025-02-15 07:03:26,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 07:03:26,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39095.53 MB 2025-02-15 07:03:27,085 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 07:03:27,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:27,086 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:03:27,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:27,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:03:27,092 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:03:27,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:27,093 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:03:27,093 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:03:50,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:50,004 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:03:50,009 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:03:50,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:50,012 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1890, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:03:50,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:03:50,013 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1890, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:04:19,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:04:19,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:04:19,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.13 seconds 2025-02-15 07:04:19,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:19,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39052.73 MB 2025-02-15 07:04:19,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45741.34 MB 2025-02-15 07:04:19,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6688.60 MB 2025-02-15 07:04:19,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53454.31 MB 2025-02-15 07:04:19,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54809.07 MB 2025-02-15 07:04:19,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1354.76 MB 2025-02-15 07:04:19,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54639.40 MB 2025-02-15 07:04:19,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:04:19,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:04:19,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 07:04:19,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:19,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45741.34 MB 2025-02-15 07:04:19,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38517.57 MB 2025-02-15 07:04:19,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7223.76 MB 2025-02-15 07:04:19,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54809.07 MB 2025-02-15 07:04:19,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69260.54 MB 2025-02-15 07:04:19,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14451.47 MB 2025-02-15 07:04:19,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65346.54 MB 2025-02-15 07:04:21,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:04:21,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:04:21,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:04:21,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38517.57 MB 2025-02-15 07:04:21,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39048.41 MB 2025-02-15 07:04:21,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:04:21,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69260.54 MB 2025-02-15 07:04:21,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45346.72 MB 2025-02-15 07:04:21,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23913.82 MB 2025-02-15 07:04:21,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43026.96 MB 2025-02-15 07:04:21,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:04:21,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:04:21,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:04:21,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39048.41 MB 2025-02-15 07:04:21,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40937.95 MB 2025-02-15 07:04:21,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:04:21,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45346.72 MB 2025-02-15 07:04:21,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46290.44 MB 2025-02-15 07:04:21,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:04:21,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42355.38 MB 2025-02-15 07:04:21,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:04:21,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:04:21,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 07:04:21,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40937.95 MB 2025-02-15 07:04:21,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43179.80 MB 2025-02-15 07:04:21,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:04:21,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46290.44 MB 2025-02-15 07:04:21,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51952.75 MB 2025-02-15 07:04:21,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:04:21,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48724.09 MB 2025-02-15 07:04:21,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:04:21,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:04:21,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:04:21,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39048.41 MB 2025-02-15 07:04:21,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43179.80 MB 2025-02-15 07:04:21,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:04:21,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45346.72 MB 2025-02-15 07:04:21,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51952.75 MB 2025-02-15 07:04:21,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:04:21,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48724.09 MB 2025-02-15 07:04:21,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:04:21,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:04:21,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:04:21,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44713.35 MB 2025-02-15 07:04:21,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45480.35 MB 2025-02-15 07:04:21,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:04:21,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51952.75 MB 2025-02-15 07:04:21,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:04:21,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:04:21,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46188.14 MB 2025-02-15 07:04:21,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:04:21,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:04:21,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:04:21,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45893.24 MB 2025-02-15 07:04:21,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46121.65 MB 2025-02-15 07:04:21,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.41 MB 2025-02-15 07:04:21,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:04:21,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:04:21,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:04:21,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46331.31 MB 2025-02-15 07:04:21,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:04:21,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:04:21,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.62 seconds 2025-02-15 07:04:21,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32467.82 MB 2025-02-15 07:04:21,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46322.50 MB 2025-02-15 07:04:21,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13854.68 MB 2025-02-15 07:04:21,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53454.31 MB 2025-02-15 07:04:21,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:04:21,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1088.42 MB 2025-02-15 07:04:21,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46331.31 MB 2025-02-15 07:04:21,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:04:21,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:04:21,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:04:21,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46322.50 MB 2025-02-15 07:04:21,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37465.03 MB 2025-02-15 07:04:21,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8857.46 MB 2025-02-15 07:04:21,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:04:21,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:04:21,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:04:21,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48828.63 MB 2025-02-15 07:04:21,924 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 07:04:21,924 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:04:21,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:04:21,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:04:21,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:04:21,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:04:21,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37465.03 MB 2025-02-15 07:04:21,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45885.81 MB 2025-02-15 07:04:21,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 07:04:21,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:04:21,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56551.80 MB 2025-02-15 07:04:21,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 07:04:21,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45885.81 MB 2025-02-15 07:04:22,087 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 07:04:22,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:04:22,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:04:22,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:04:22,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:04:22,094 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:04:22,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:04:22,095 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:04:22,095 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:05:08,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:08,414 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:05:08,419 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:05:08,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:08,423 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 443, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:05:08,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:08,424 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 443, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:05:15,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:05:15,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:05:15,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.82 seconds 2025-02-15 07:05:15,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:15,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28969.80 MB 2025-02-15 07:05:15,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30537.56 MB 2025-02-15 07:05:15,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1567.75 MB 2025-02-15 07:05:15,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64923.63 MB 2025-02-15 07:05:15,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-15 07:05:15,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25591.55 MB 2025-02-15 07:05:15,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39347.14 MB 2025-02-15 07:05:15,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:05:15,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:05:15,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 07:05:15,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:15,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30537.56 MB 2025-02-15 07:05:15,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30996.12 MB 2025-02-15 07:05:15,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.56 MB 2025-02-15 07:05:15,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39332.09 MB 2025-02-15 07:05:15,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41456.50 MB 2025-02-15 07:05:15,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2124.41 MB 2025-02-15 07:05:15,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37874.59 MB 2025-02-15 07:05:17,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:05:17,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:05:17,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:05:17,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30996.12 MB 2025-02-15 07:05:17,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31526.96 MB 2025-02-15 07:05:17,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:05:17,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41456.50 MB 2025-02-15 07:05:17,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40512.78 MB 2025-02-15 07:05:17,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 07:05:17,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.51 MB 2025-02-15 07:05:17,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:05:17,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:05:17,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:05:17,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31526.96 MB 2025-02-15 07:05:17,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33416.50 MB 2025-02-15 07:05:17,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:05:17,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40512.78 MB 2025-02-15 07:05:17,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40512.78 MB 2025-02-15 07:05:17,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:17,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34833.92 MB 2025-02-15 07:05:17,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:05:17,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:05:17,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:05:17,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33416.50 MB 2025-02-15 07:05:17,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35658.35 MB 2025-02-15 07:05:17,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:05:17,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40512.78 MB 2025-02-15 07:05:17,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44287.66 MB 2025-02-15 07:05:17,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 07:05:17,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41202.63 MB 2025-02-15 07:05:17,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:05:17,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:05:17,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:05:17,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31526.96 MB 2025-02-15 07:05:17,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35658.35 MB 2025-02-15 07:05:17,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:05:17,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40512.78 MB 2025-02-15 07:05:17,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44287.66 MB 2025-02-15 07:05:17,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 07:05:17,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41202.63 MB 2025-02-15 07:05:17,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:05:17,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:05:17,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 07:05:17,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37191.89 MB 2025-02-15 07:05:17,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37958.90 MB 2025-02-15 07:05:17,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:05:17,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44287.66 MB 2025-02-15 07:05:17,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44698.70 MB 2025-02-15 07:05:17,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:05:17,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38666.68 MB 2025-02-15 07:05:17,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:05:17,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:05:17,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:05:17,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38371.78 MB 2025-02-15 07:05:17,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38599.91 MB 2025-02-15 07:05:17,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-15 07:05:17,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44698.70 MB 2025-02-15 07:05:17,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44698.70 MB 2025-02-15 07:05:17,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:17,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38802.39 MB 2025-02-15 07:05:17,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:05:17,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:05:17,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.22 seconds 2025-02-15 07:05:17,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.36 MB 2025-02-15 07:05:17,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38800.98 MB 2025-02-15 07:05:17,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11374.63 MB 2025-02-15 07:05:17,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64923.63 MB 2025-02-15 07:05:17,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44698.70 MB 2025-02-15 07:05:17,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20224.93 MB 2025-02-15 07:05:17,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38802.39 MB 2025-02-15 07:05:17,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:05:17,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:05:17,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:05:17,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38800.98 MB 2025-02-15 07:05:17,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32430.20 MB 2025-02-15 07:05:17,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6370.78 MB 2025-02-15 07:05:17,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44698.70 MB 2025-02-15 07:05:17,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44698.70 MB 2025-02-15 07:05:17,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:17,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41312.65 MB 2025-02-15 07:05:17,931 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:05:17,931 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:05:17,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:05:17,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:05:17,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:05:17,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:17,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32430.20 MB 2025-02-15 07:05:17,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40869.23 MB 2025-02-15 07:05:17,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:05:17,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44698.70 MB 2025-02-15 07:05:17,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53089.40 MB 2025-02-15 07:05:17,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:05:17,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40869.23 MB 2025-02-15 07:05:18,144 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:05:18,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:18,147 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:05:18,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:18,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:05:18,156 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:05:18,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:18,158 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:05:18,158 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:05:32,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:32,938 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:05:32,943 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:05:32,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:32,947 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:05:32,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:32,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:05:48,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:05:48,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:05:48,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.62 seconds 2025-02-15 07:05:48,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:48,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32865.01 MB 2025-02-15 07:05:48,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36411.29 MB 2025-02-15 07:05:48,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3546.28 MB 2025-02-15 07:05:48,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65674.41 MB 2025-02-15 07:05:48,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43297.80 MB 2025-02-15 07:05:48,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22376.61 MB 2025-02-15 07:05:48,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45280.78 MB 2025-02-15 07:05:48,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:05:48,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:05:48,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:05:48,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:48,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36411.29 MB 2025-02-15 07:05:48,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33902.19 MB 2025-02-15 07:05:48,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2509.10 MB 2025-02-15 07:05:48,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43297.80 MB 2025-02-15 07:05:48,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52435.09 MB 2025-02-15 07:05:48,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9137.29 MB 2025-02-15 07:05:48,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47646.20 MB 2025-02-15 07:05:50,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:05:50,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:05:50,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:05:50,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33902.19 MB 2025-02-15 07:05:50,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34433.03 MB 2025-02-15 07:05:50,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:05:50,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52435.09 MB 2025-02-15 07:05:50,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41875.93 MB 2025-02-15 07:05:50,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10559.16 MB 2025-02-15 07:05:50,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38411.58 MB 2025-02-15 07:05:50,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:05:50,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:05:50,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:05:50,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34433.03 MB 2025-02-15 07:05:50,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36322.56 MB 2025-02-15 07:05:50,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:05:50,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41875.93 MB 2025-02-15 07:05:50,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41875.93 MB 2025-02-15 07:05:50,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:50,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37739.99 MB 2025-02-15 07:05:50,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:05:50,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:05:50,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:05:50,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36322.56 MB 2025-02-15 07:05:50,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38564.42 MB 2025-02-15 07:05:50,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:05:50,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41875.93 MB 2025-02-15 07:05:50,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46594.52 MB 2025-02-15 07:05:50,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 07:05:50,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44108.70 MB 2025-02-15 07:05:50,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:05:50,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:05:50,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:05:50,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34433.03 MB 2025-02-15 07:05:50,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38564.42 MB 2025-02-15 07:05:50,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:05:50,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41875.93 MB 2025-02-15 07:05:50,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46594.52 MB 2025-02-15 07:05:50,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 07:05:50,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44108.70 MB 2025-02-15 07:05:50,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:05:50,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:05:50,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:05:50,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40097.96 MB 2025-02-15 07:05:50,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40864.96 MB 2025-02-15 07:05:50,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:05:50,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46594.52 MB 2025-02-15 07:05:50,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47005.56 MB 2025-02-15 07:05:50,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:05:50,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41572.75 MB 2025-02-15 07:05:50,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:05:50,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:05:50,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:05:50,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41277.85 MB 2025-02-15 07:05:50,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41504.33 MB 2025-02-15 07:05:50,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.48 MB 2025-02-15 07:05:50,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47005.56 MB 2025-02-15 07:05:50,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47005.56 MB 2025-02-15 07:05:50,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:50,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41724.54 MB 2025-02-15 07:05:50,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:05:50,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:05:50,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.03 seconds 2025-02-15 07:05:50,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:50,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29373.96 MB 2025-02-15 07:05:50,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41704.35 MB 2025-02-15 07:05:50,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12330.39 MB 2025-02-15 07:05:50,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65674.41 MB 2025-02-15 07:05:50,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47005.56 MB 2025-02-15 07:05:50,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18668.85 MB 2025-02-15 07:05:50,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41724.54 MB 2025-02-15 07:05:51,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:05:51,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:05:51,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:05:51,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:51,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41704.35 MB 2025-02-15 07:05:51,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34362.50 MB 2025-02-15 07:05:51,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.84 MB 2025-02-15 07:05:51,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47005.56 MB 2025-02-15 07:05:51,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47005.56 MB 2025-02-15 07:05:51,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:05:51,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44203.64 MB 2025-02-15 07:05:51,265 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 07:05:51,265 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:05:51,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:05:51,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:05:51,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:05:51,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:05:51,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34362.50 MB 2025-02-15 07:05:51,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42757.72 MB 2025-02-15 07:05:51,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 07:05:51,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47005.56 MB 2025-02-15 07:05:51,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55352.23 MB 2025-02-15 07:05:51,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 07:05:51,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42757.72 MB 2025-02-15 07:05:51,432 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 07:05:51,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:51,434 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:05:51,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:51,435 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:05:51,439 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:05:51,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:05:51,440 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:05:51,440 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:07:34,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:34,572 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:07:34,578 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:07:34,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:34,582 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 222, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:07:34,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:34,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 222, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:07:38,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:07:38,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:07:38,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.41 seconds 2025-02-15 07:07:38,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:38,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27429.84 MB 2025-02-15 07:07:38,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28215.49 MB 2025-02-15 07:07:38,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 785.65 MB 2025-02-15 07:07:38,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63698.89 MB 2025-02-15 07:07:38,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:38,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27432.85 MB 2025-02-15 07:07:38,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37127.70 MB 2025-02-15 07:07:38,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:07:38,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:07:38,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:07:38,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:38,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28215.49 MB 2025-02-15 07:07:38,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28596.06 MB 2025-02-15 07:07:38,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 380.58 MB 2025-02-15 07:07:38,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:38,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:38,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:38,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31333.71 MB 2025-02-15 07:07:39,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:07:39,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:07:39,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.05 seconds 2025-02-15 07:07:39,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28596.06 MB 2025-02-15 07:07:39,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.68 MB 2025-02-15 07:07:39,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-15 07:07:39,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:39,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:39,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32850.65 MB 2025-02-15 07:07:39,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:07:39,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:07:39,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:07:39,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28890.68 MB 2025-02-15 07:07:39,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29939.12 MB 2025-02-15 07:07:39,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-15 07:07:39,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:39,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:39,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30725.79 MB 2025-02-15 07:07:39,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:07:39,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:07:39,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 07:07:39,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29939.12 MB 2025-02-15 07:07:39,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.38 MB 2025-02-15 07:07:39,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.26 MB 2025-02-15 07:07:39,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:39,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:39,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.42 MB 2025-02-15 07:07:39,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:07:39,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:07:39,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 07:07:39,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28890.68 MB 2025-02-15 07:07:39,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.38 MB 2025-02-15 07:07:39,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2292.70 MB 2025-02-15 07:07:39,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:39,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36266.05 MB 2025-02-15 07:07:39,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.42 MB 2025-02-15 07:07:39,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:07:39,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:07:39,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:07:39,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32034.49 MB 2025-02-15 07:07:39,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32460.18 MB 2025-02-15 07:07:39,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.69 MB 2025-02-15 07:07:39,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36266.05 MB 2025-02-15 07:07:39,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-15 07:07:39,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-15 07:07:39,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32853.00 MB 2025-02-15 07:07:39,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:07:39,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:07:39,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:07:39,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32689.34 MB 2025-02-15 07:07:39,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32909.19 MB 2025-02-15 07:07:39,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.86 MB 2025-02-15 07:07:39,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-15 07:07:39,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-15 07:07:39,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.33 MB 2025-02-15 07:07:39,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:07:39,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:07:39,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.73 seconds 2025-02-15 07:07:39,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26656.37 MB 2025-02-15 07:07:39,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33110.05 MB 2025-02-15 07:07:39,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6453.67 MB 2025-02-15 07:07:39,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63698.89 MB 2025-02-15 07:07:39,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-15 07:07:39,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27206.35 MB 2025-02-15 07:07:39,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33110.05 MB 2025-02-15 07:07:39,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:07:39,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:07:39,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 07:07:39,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27806.04 MB 2025-02-15 07:07:39,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30816.76 MB 2025-02-15 07:07:39,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.72 MB 2025-02-15 07:07:39,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-15 07:07:39,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-15 07:07:39,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:07:39,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31117.80 MB 2025-02-15 07:07:39,600 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 07:07:39,600 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:07:39,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:07:39,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:07:39,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:07:39,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:07:39,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30816.76 MB 2025-02-15 07:07:39,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39247.16 MB 2025-02-15 07:07:39,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 07:07:39,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-15 07:07:39,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44872.76 MB 2025-02-15 07:07:39,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 07:07:39,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39247.16 MB 2025-02-15 07:07:39,767 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 07:07:39,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:39,768 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:07:39,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:39,769 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:07:39,774 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:07:39,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:07:39,775 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:07:39,775 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:08:39,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:08:39,603 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:08:39,610 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:08:39,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:08:39,616 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2451, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:08:39,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:08:39,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2451, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:09:17,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:09:17,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:09:17,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.80 seconds 2025-02-15 07:09:17,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:17,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42962.25 MB 2025-02-15 07:09:17,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51636.20 MB 2025-02-15 07:09:17,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8673.95 MB 2025-02-15 07:09:17,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70336.38 MB 2025-02-15 07:09:17,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61314.43 MB 2025-02-15 07:09:17,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9021.95 MB 2025-02-15 07:09:17,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60587.34 MB 2025-02-15 07:09:17,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:09:17,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:09:17,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 07:09:17,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:17,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51636.20 MB 2025-02-15 07:09:17,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41434.41 MB 2025-02-15 07:09:17,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10201.78 MB 2025-02-15 07:09:17,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61314.43 MB 2025-02-15 07:09:17,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77301.02 MB 2025-02-15 07:09:17,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15986.59 MB 2025-02-15 07:09:17,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 75677.08 MB 2025-02-15 07:09:19,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:09:19,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:09:19,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:09:19,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41434.41 MB 2025-02-15 07:09:19,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41965.26 MB 2025-02-15 07:09:19,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:09:19,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77301.02 MB 2025-02-15 07:09:19,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48299.51 MB 2025-02-15 07:09:19,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29001.52 MB 2025-02-15 07:09:19,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45943.80 MB 2025-02-15 07:09:19,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:09:19,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:09:19,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:09:19,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41965.26 MB 2025-02-15 07:09:19,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43854.72 MB 2025-02-15 07:09:19,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-15 07:09:19,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48299.51 MB 2025-02-15 07:09:19,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48299.51 MB 2025-02-15 07:09:19,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:09:19,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45272.15 MB 2025-02-15 07:09:19,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:09:19,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:09:19,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:09:19,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43854.72 MB 2025-02-15 07:09:19,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46096.58 MB 2025-02-15 07:09:19,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:09:19,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48299.51 MB 2025-02-15 07:09:19,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53961.82 MB 2025-02-15 07:09:19,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:09:19,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51640.86 MB 2025-02-15 07:09:19,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:09:19,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:09:19,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:09:19,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41965.26 MB 2025-02-15 07:09:19,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46096.58 MB 2025-02-15 07:09:19,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-15 07:09:19,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48299.51 MB 2025-02-15 07:09:19,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53961.82 MB 2025-02-15 07:09:19,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:09:19,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51640.86 MB 2025-02-15 07:09:19,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:09:19,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:09:19,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:09:19,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47630.12 MB 2025-02-15 07:09:19,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48397.12 MB 2025-02-15 07:09:19,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:09:19,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53961.82 MB 2025-02-15 07:09:19,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54374.96 MB 2025-02-15 07:09:19,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:09:19,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49104.91 MB 2025-02-15 07:09:19,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:09:19,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:09:19,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:09:19,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48810.01 MB 2025-02-15 07:09:19,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49038.17 MB 2025-02-15 07:09:19,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-15 07:09:19,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54374.96 MB 2025-02-15 07:09:19,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54374.96 MB 2025-02-15 07:09:19,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:09:19,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49256.70 MB 2025-02-15 07:09:19,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:09:19,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:09:19,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.33 seconds 2025-02-15 07:09:19,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:19,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34422.58 MB 2025-02-15 07:09:19,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49239.03 MB 2025-02-15 07:09:19,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14816.45 MB 2025-02-15 07:09:19,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61794.68 MB 2025-02-15 07:09:19,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54374.96 MB 2025-02-15 07:09:19,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7419.72 MB 2025-02-15 07:09:19,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49256.70 MB 2025-02-15 07:09:20,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:09:20,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:09:20,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:09:20,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:20,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49239.03 MB 2025-02-15 07:09:20,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39423.00 MB 2025-02-15 07:09:20,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9816.03 MB 2025-02-15 07:09:20,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54374.96 MB 2025-02-15 07:09:20,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54374.96 MB 2025-02-15 07:09:20,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:09:20,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51747.93 MB 2025-02-15 07:09:20,244 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 07:09:20,244 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:09:20,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:09:20,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:09:20,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:09:20,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:09:20,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39423.00 MB 2025-02-15 07:09:20,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47853.15 MB 2025-02-15 07:09:20,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.16 MB 2025-02-15 07:09:20,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54374.96 MB 2025-02-15 07:09:20,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58565.07 MB 2025-02-15 07:09:20,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 07:09:20,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47853.15 MB 2025-02-15 07:09:20,409 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 07:09:20,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:09:20,410 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:09:20,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:09:20,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:09:20,416 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:09:20,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:09:20,417 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:09:20,417 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:10:07,787 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:07,787 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:10:07,795 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:10:07,802 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:07,802 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:10:07,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:07,804 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:10:25,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:10:25,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:10:25,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.40 seconds 2025-02-15 07:10:25,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:25,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33596.66 MB 2025-02-15 07:10:25,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37514.27 MB 2025-02-15 07:10:25,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3917.61 MB 2025-02-15 07:10:25,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66945.29 MB 2025-02-15 07:10:25,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43666.90 MB 2025-02-15 07:10:25,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23278.39 MB 2025-02-15 07:10:25,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46465.42 MB 2025-02-15 07:10:25,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:10:25,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:10:25,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:10:25,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:25,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37514.27 MB 2025-02-15 07:10:25,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34447.00 MB 2025-02-15 07:10:25,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3067.27 MB 2025-02-15 07:10:25,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43666.90 MB 2025-02-15 07:10:25,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52403.63 MB 2025-02-15 07:10:25,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8736.74 MB 2025-02-15 07:10:25,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49057.20 MB 2025-02-15 07:10:27,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:10:27,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:10:27,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 07:10:27,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34447.00 MB 2025-02-15 07:10:27,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34977.84 MB 2025-02-15 07:10:27,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:10:27,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52403.63 MB 2025-02-15 07:10:27,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41162.90 MB 2025-02-15 07:10:27,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11240.73 MB 2025-02-15 07:10:27,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38956.39 MB 2025-02-15 07:10:27,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:10:27,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:10:27,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:10:27,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34977.84 MB 2025-02-15 07:10:27,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36867.38 MB 2025-02-15 07:10:27,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:10:27,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41162.90 MB 2025-02-15 07:10:27,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42106.62 MB 2025-02-15 07:10:27,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:10:27,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38284.81 MB 2025-02-15 07:10:27,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:10:27,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:10:27,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:10:27,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36867.38 MB 2025-02-15 07:10:27,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39109.23 MB 2025-02-15 07:10:27,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:10:27,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42106.62 MB 2025-02-15 07:10:27,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47768.93 MB 2025-02-15 07:10:27,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:10:27,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44653.51 MB 2025-02-15 07:10:27,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:10:27,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:10:27,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:10:27,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34977.84 MB 2025-02-15 07:10:27,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39109.23 MB 2025-02-15 07:10:27,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:10:27,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41162.90 MB 2025-02-15 07:10:27,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47768.93 MB 2025-02-15 07:10:27,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:10:27,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44653.51 MB 2025-02-15 07:10:27,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:10:27,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:10:27,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:10:27,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40642.78 MB 2025-02-15 07:10:27,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41409.78 MB 2025-02-15 07:10:27,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:10:27,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47768.93 MB 2025-02-15 07:10:27,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 07:10:27,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:10:27,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42117.57 MB 2025-02-15 07:10:27,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:10:27,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:10:27,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:10:27,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41822.67 MB 2025-02-15 07:10:27,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42048.39 MB 2025-02-15 07:10:27,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.73 MB 2025-02-15 07:10:27,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 07:10:27,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 07:10:27,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:10:27,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42266.83 MB 2025-02-15 07:10:27,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:10:27,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:10:27,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.85 seconds 2025-02-15 07:10:27,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29739.79 MB 2025-02-15 07:10:27,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42249.25 MB 2025-02-15 07:10:27,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12509.46 MB 2025-02-15 07:10:27,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66945.29 MB 2025-02-15 07:10:27,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 07:10:27,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18765.32 MB 2025-02-15 07:10:27,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42266.83 MB 2025-02-15 07:10:27,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:10:27,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:10:27,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:10:27,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:27,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42249.25 MB 2025-02-15 07:10:27,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34733.08 MB 2025-02-15 07:10:27,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7516.17 MB 2025-02-15 07:10:27,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 07:10:27,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 07:10:27,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:10:27,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44752.00 MB 2025-02-15 07:10:27,992 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 07:10:27,992 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:10:28,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:10:28,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:10:28,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:10:28,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:10:28,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34733.08 MB 2025-02-15 07:10:28,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43142.38 MB 2025-02-15 07:10:28,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 07:10:28,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 07:10:28,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56539.22 MB 2025-02-15 07:10:28,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 07:10:28,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43142.38 MB 2025-02-15 07:10:28,159 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 07:10:28,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:28,161 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:10:28,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:28,162 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:10:28,166 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:10:28,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:10:28,167 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:10:28,167 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:11:48,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:11:48,339 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:11:48,344 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:11:48,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:11:48,349 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:11:48,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:11:48,350 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:12:06,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:12:06,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:12:06,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.55 seconds 2025-02-15 07:12:06,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:06,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34251.67 MB 2025-02-15 07:12:06,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38502.60 MB 2025-02-15 07:12:06,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-15 07:12:06,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64898.47 MB 2025-02-15 07:12:06,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48167.39 MB 2025-02-15 07:12:06,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16731.08 MB 2025-02-15 07:12:06,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47346.92 MB 2025-02-15 07:12:06,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:12:06,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:12:06,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:12:06,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:06,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38502.60 MB 2025-02-15 07:12:06,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34936.73 MB 2025-02-15 07:12:06,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3565.87 MB 2025-02-15 07:12:06,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48167.39 MB 2025-02-15 07:12:06,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53202.65 MB 2025-02-15 07:12:06,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5035.26 MB 2025-02-15 07:12:06,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48501.36 MB 2025-02-15 07:12:08,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:12:08,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:12:08,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:12:08,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:08,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34936.73 MB 2025-02-15 07:12:08,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35467.57 MB 2025-02-15 07:12:08,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:12:08,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53202.65 MB 2025-02-15 07:12:08,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-15 07:12:08,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7161.77 MB 2025-02-15 07:12:08,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39446.64 MB 2025-02-15 07:12:08,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:12:08,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:12:08,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:12:08,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:08,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35467.57 MB 2025-02-15 07:12:08,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37357.10 MB 2025-02-15 07:12:08,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:12:08,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-15 07:12:08,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-15 07:12:08,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:12:08,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38774.53 MB 2025-02-15 07:12:09,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:12:09,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:12:09,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:12:09,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37357.10 MB 2025-02-15 07:12:09,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39598.96 MB 2025-02-15 07:12:09,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:12:09,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-15 07:12:09,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48872.03 MB 2025-02-15 07:12:09,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:12:09,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45143.24 MB 2025-02-15 07:12:09,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:12:09,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:12:09,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:12:09,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35467.57 MB 2025-02-15 07:12:09,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39598.96 MB 2025-02-15 07:12:09,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:12:09,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-15 07:12:09,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48872.03 MB 2025-02-15 07:12:09,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:12:09,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45143.24 MB 2025-02-15 07:12:09,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:12:09,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:12:09,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:12:09,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41132.50 MB 2025-02-15 07:12:09,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41899.50 MB 2025-02-15 07:12:09,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:12:09,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48872.03 MB 2025-02-15 07:12:09,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49283.07 MB 2025-02-15 07:12:09,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:12:09,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42607.29 MB 2025-02-15 07:12:09,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:12:09,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:12:09,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:12:09,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42312.39 MB 2025-02-15 07:12:09,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42542.43 MB 2025-02-15 07:12:09,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.04 MB 2025-02-15 07:12:09,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49283.07 MB 2025-02-15 07:12:09,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49283.07 MB 2025-02-15 07:12:09,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:12:09,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42761.21 MB 2025-02-15 07:12:09,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:12:09,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:12:09,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.94 seconds 2025-02-15 07:12:09,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30067.29 MB 2025-02-15 07:12:09,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42743.51 MB 2025-02-15 07:12:09,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12676.22 MB 2025-02-15 07:12:09,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64898.47 MB 2025-02-15 07:12:09,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49283.07 MB 2025-02-15 07:12:09,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15615.39 MB 2025-02-15 07:12:09,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42761.21 MB 2025-02-15 07:12:09,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:12:09,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:12:09,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:12:09,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42743.51 MB 2025-02-15 07:12:09,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35071.14 MB 2025-02-15 07:12:09,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7672.37 MB 2025-02-15 07:12:09,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49283.07 MB 2025-02-15 07:12:09,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49283.07 MB 2025-02-15 07:12:09,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:12:09,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45255.17 MB 2025-02-15 07:12:09,574 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:12:09,575 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:12:09,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:12:09,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:12:09,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:12:09,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:09,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35071.14 MB 2025-02-15 07:12:09,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43510.16 MB 2025-02-15 07:12:09,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:12:09,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49283.07 MB 2025-02-15 07:12:09,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57673.78 MB 2025-02-15 07:12:09,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:12:09,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43510.16 MB 2025-02-15 07:12:09,743 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:12:09,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:09,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:12:09,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:09,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:12:09,750 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:12:09,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:09,751 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:12:09,751 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:12:19,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:19,639 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:12:19,644 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:12:19,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:19,648 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2012, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:12:19,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:19,649 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2012, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:12:51,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:12:51,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:12:51,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.51 seconds 2025-02-15 07:12:51,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:51,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39902.85 MB 2025-02-15 07:12:51,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47023.20 MB 2025-02-15 07:12:51,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7120.36 MB 2025-02-15 07:12:51,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70258.79 MB 2025-02-15 07:12:51,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55232.69 MB 2025-02-15 07:12:51,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15026.09 MB 2025-02-15 07:12:51,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55942.50 MB 2025-02-15 07:12:51,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:12:51,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:12:51,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 07:12:51,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:51,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47023.20 MB 2025-02-15 07:12:51,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39151.81 MB 2025-02-15 07:12:51,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7871.39 MB 2025-02-15 07:12:51,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55232.69 MB 2025-02-15 07:12:51,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70271.37 MB 2025-02-15 07:12:51,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15038.68 MB 2025-02-15 07:12:51,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67507.32 MB 2025-02-15 07:12:53,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:12:53,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:12:53,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:12:53,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39151.81 MB 2025-02-15 07:12:53,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39682.65 MB 2025-02-15 07:12:53,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:12:53,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70271.37 MB 2025-02-15 07:12:53,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45346.72 MB 2025-02-15 07:12:53,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24924.65 MB 2025-02-15 07:12:53,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43661.20 MB 2025-02-15 07:12:53,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:12:53,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:12:53,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:12:53,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39682.65 MB 2025-02-15 07:12:53,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41572.19 MB 2025-02-15 07:12:53,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:12:53,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45346.72 MB 2025-02-15 07:12:53,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47234.15 MB 2025-02-15 07:12:53,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 07:12:53,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42989.62 MB 2025-02-15 07:12:53,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:12:53,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:12:53,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 07:12:53,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41572.19 MB 2025-02-15 07:12:53,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43814.04 MB 2025-02-15 07:12:53,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:12:53,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47234.15 MB 2025-02-15 07:12:53,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52896.46 MB 2025-02-15 07:12:53,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:12:53,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49358.33 MB 2025-02-15 07:12:53,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:12:53,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:12:53,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:12:53,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39682.65 MB 2025-02-15 07:12:53,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43814.04 MB 2025-02-15 07:12:53,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:12:53,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45346.72 MB 2025-02-15 07:12:53,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52896.46 MB 2025-02-15 07:12:53,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 07:12:53,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49358.33 MB 2025-02-15 07:12:53,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:12:53,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:12:53,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:12:53,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45347.59 MB 2025-02-15 07:12:53,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46114.59 MB 2025-02-15 07:12:53,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:12:53,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52896.46 MB 2025-02-15 07:12:53,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53307.51 MB 2025-02-15 07:12:53,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:12:53,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46822.38 MB 2025-02-15 07:12:53,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:12:53,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:12:53,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:12:53,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46527.48 MB 2025-02-15 07:12:53,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46755.53 MB 2025-02-15 07:12:53,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-15 07:12:53,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53307.51 MB 2025-02-15 07:12:53,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53307.51 MB 2025-02-15 07:12:53,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:12:53,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46975.48 MB 2025-02-15 07:12:53,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:12:53,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:12:53,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.00 seconds 2025-02-15 07:12:53,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32892.88 MB 2025-02-15 07:12:53,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46956.38 MB 2025-02-15 07:12:53,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14063.50 MB 2025-02-15 07:12:53,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70258.79 MB 2025-02-15 07:12:53,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53307.51 MB 2025-02-15 07:12:53,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16951.28 MB 2025-02-15 07:12:53,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46975.48 MB 2025-02-15 07:12:53,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:12:53,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:12:53,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:12:53,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46956.38 MB 2025-02-15 07:12:53,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37888.31 MB 2025-02-15 07:12:53,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9068.07 MB 2025-02-15 07:12:53,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53307.51 MB 2025-02-15 07:12:53,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53307.51 MB 2025-02-15 07:12:53,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:12:53,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49460.98 MB 2025-02-15 07:12:53,945 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 07:12:53,945 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:12:53,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:12:53,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:12:53,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:12:53,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:12:53,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37888.31 MB 2025-02-15 07:12:53,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46303.26 MB 2025-02-15 07:12:53,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 07:12:53,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53307.51 MB 2025-02-15 07:12:53,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61675.14 MB 2025-02-15 07:12:53,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 07:12:53,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46303.26 MB 2025-02-15 07:12:54,111 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 07:12:54,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:54,113 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:12:54,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:54,114 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:12:54,119 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:12:54,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:12:54,120 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:12:54,120 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:13:02,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:02,456 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:13:02,461 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:13:02,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:02,464 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:13:02,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:02,465 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:13:04,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:13:04,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:13:04,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-15 07:13:04,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:04,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26956.00 MB 2025-02-15 07:13:04,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27501.00 MB 2025-02-15 07:13:04,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-15 07:13:04,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70042.78 MB 2025-02-15 07:13:04,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:04,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33539.75 MB 2025-02-15 07:13:04,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36427.38 MB 2025-02-15 07:13:04,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:13:04,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:13:04,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:13:04,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:04,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27501.00 MB 2025-02-15 07:13:04,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27765.05 MB 2025-02-15 07:13:04,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.05 MB 2025-02-15 07:13:04,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:04,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:04,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:04,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29664.16 MB 2025-02-15 07:13:05,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:13:05,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:13:05,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.75 seconds 2025-02-15 07:13:05,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27765.05 MB 2025-02-15 07:13:05,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27969.43 MB 2025-02-15 07:13:05,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.37 MB 2025-02-15 07:13:05,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:05,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:05,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:05,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31934.70 MB 2025-02-15 07:13:05,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:13:05,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:13:05,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:13:05,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27969.36 MB 2025-02-15 07:13:05,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28696.65 MB 2025-02-15 07:13:05,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 727.29 MB 2025-02-15 07:13:05,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:05,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:05,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:05,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29242.37 MB 2025-02-15 07:13:05,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:13:05,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:13:05,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:13:05,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28696.65 MB 2025-02-15 07:13:05,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29559.81 MB 2025-02-15 07:13:05,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 863.15 MB 2025-02-15 07:13:05,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:05,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:05,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:05,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31694.32 MB 2025-02-15 07:13:05,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:13:05,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:13:05,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:13:05,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27969.36 MB 2025-02-15 07:13:05,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29559.81 MB 2025-02-15 07:13:05,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1590.45 MB 2025-02-15 07:13:05,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:05,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:13:05,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:05,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31694.32 MB 2025-02-15 07:13:05,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:13:05,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:13:05,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:13:05,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30150.22 MB 2025-02-15 07:13:05,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30445.52 MB 2025-02-15 07:13:05,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 295.30 MB 2025-02-15 07:13:05,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:13:05,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36654.02 MB 2025-02-15 07:13:05,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 07:13:05,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30727.80 MB 2025-02-15 07:13:05,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:13:05,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:13:05,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:13:05,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30604.49 MB 2025-02-15 07:13:05,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30809.33 MB 2025-02-15 07:13:05,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.84 MB 2025-02-15 07:13:05,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36654.02 MB 2025-02-15 07:13:05,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36658.22 MB 2025-02-15 07:13:05,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 07:13:05,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30829.10 MB 2025-02-15 07:13:05,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:13:05,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:13:05,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.35 seconds 2025-02-15 07:13:05,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:05,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26419.46 MB 2025-02-15 07:13:05,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31010.38 MB 2025-02-15 07:13:05,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4590.92 MB 2025-02-15 07:13:05,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70042.78 MB 2025-02-15 07:13:05,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36658.22 MB 2025-02-15 07:13:05,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33384.56 MB 2025-02-15 07:13:05,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31010.38 MB 2025-02-15 07:13:06,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:13:06,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:13:06,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:13:06,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:06,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31010.38 MB 2025-02-15 07:13:06,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30261.98 MB 2025-02-15 07:13:06,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -748.40 MB 2025-02-15 07:13:06,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36658.22 MB 2025-02-15 07:13:06,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36658.22 MB 2025-02-15 07:13:06,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:13:06,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31914.47 MB 2025-02-15 07:13:06,101 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 07:13:06,102 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:13:06,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:13:06,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:13:06,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:13:06,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:13:06,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30261.98 MB 2025-02-15 07:13:06,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38700.81 MB 2025-02-15 07:13:06,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 07:13:06,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36658.22 MB 2025-02-15 07:13:06,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45046.82 MB 2025-02-15 07:13:06,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 07:13:06,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38700.81 MB 2025-02-15 07:13:06,267 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 07:13:06,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:06,269 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:13:06,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:06,269 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:13:06,274 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:13:06,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:13:06,275 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:13:06,275 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:14:12,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:12,249 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:14:12,254 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:14:12,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:12,258 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:14:12,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:12,259 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:14:14,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:14:14,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:14:14,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.16 seconds 2025-02-15 07:14:14,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:14,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26830.58 MB 2025-02-15 07:14:14,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27311.87 MB 2025-02-15 07:14:14,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-15 07:14:14,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53435.43 MB 2025-02-15 07:14:14,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:14,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17878.22 MB 2025-02-15 07:14:14,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36301.95 MB 2025-02-15 07:14:14,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:14:14,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:14:14,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:14:14,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:14,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27311.87 MB 2025-02-15 07:14:14,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27502.92 MB 2025-02-15 07:14:14,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-15 07:14:14,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:14,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:14,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:14,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29137.92 MB 2025-02-15 07:14:15,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:14:15,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:14:15,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.62 seconds 2025-02-15 07:14:15,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27502.92 MB 2025-02-15 07:14:15,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27675.45 MB 2025-02-15 07:14:15,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-15 07:14:15,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:15,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:15,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:15,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31672.57 MB 2025-02-15 07:14:15,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:14:15,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:14:15,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:14:15,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27675.38 MB 2025-02-15 07:14:15,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28289.33 MB 2025-02-15 07:14:15,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-15 07:14:15,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:15,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:15,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:15,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28750.00 MB 2025-02-15 07:14:15,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:14:15,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:14:15,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:14:15,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28289.33 MB 2025-02-15 07:14:15,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.98 MB 2025-02-15 07:14:15,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-15 07:14:15,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:15,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:15,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:15,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30819.83 MB 2025-02-15 07:14:15,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:14:15,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:14:15,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:14:15,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27675.38 MB 2025-02-15 07:14:15,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.98 MB 2025-02-15 07:14:15,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-15 07:14:15,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:15,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:14:15,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:15,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30819.83 MB 2025-02-15 07:14:15,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:14:15,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:14:15,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:14:15,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29516.38 MB 2025-02-15 07:14:15,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29765.66 MB 2025-02-15 07:14:15,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-15 07:14:15,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:14:15,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35685.14 MB 2025-02-15 07:14:15,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 127.93 MB 2025-02-15 07:14:15,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30008.21 MB 2025-02-15 07:14:15,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:14:15,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:14:15,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:14:15,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29899.85 MB 2025-02-15 07:14:15,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30104.40 MB 2025-02-15 07:14:15,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.55 MB 2025-02-15 07:14:15,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35685.14 MB 2025-02-15 07:14:15,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35689.33 MB 2025-02-15 07:14:15,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 07:14:15,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30104.40 MB 2025-02-15 07:14:15,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:14:15,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:14:15,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.94 seconds 2025-02-15 07:14:15,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26356.74 MB 2025-02-15 07:14:15,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30305.15 MB 2025-02-15 07:14:15,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3948.41 MB 2025-02-15 07:14:15,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53435.43 MB 2025-02-15 07:14:15,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35689.33 MB 2025-02-15 07:14:15,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17746.10 MB 2025-02-15 07:14:15,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30305.15 MB 2025-02-15 07:14:15,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:14:15,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:14:15,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:14:15,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30305.15 MB 2025-02-15 07:14:15,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30081.53 MB 2025-02-15 07:14:15,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -223.62 MB 2025-02-15 07:14:15,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35689.33 MB 2025-02-15 07:14:15,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35689.33 MB 2025-02-15 07:14:15,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:14:15,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31709.45 MB 2025-02-15 07:14:15,491 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 07:14:15,491 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:14:15,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:14:15,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:14:15,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:14:15,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:14:15,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30081.53 MB 2025-02-15 07:14:15,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38507.71 MB 2025-02-15 07:14:15,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 07:14:15,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35689.33 MB 2025-02-15 07:14:15,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44065.36 MB 2025-02-15 07:14:15,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 07:14:15,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38507.71 MB 2025-02-15 07:14:15,661 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 07:14:15,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:15,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:14:15,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:15,664 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:14:15,668 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:14:15,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:14:15,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:14:15,670 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:15:14,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:14,371 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:15:14,378 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:15:14,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:14,385 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1683, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:15:14,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:14,387 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1683, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:15:40,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:15:40,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:15:40,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.08 seconds 2025-02-15 07:15:40,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:40,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37610.32 MB 2025-02-15 07:15:40,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43566.37 MB 2025-02-15 07:15:40,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5956.04 MB 2025-02-15 07:15:40,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52441.38 MB 2025-02-15 07:15:40,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54079.26 MB 2025-02-15 07:15:40,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1637.88 MB 2025-02-15 07:15:40,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52517.51 MB 2025-02-15 07:15:40,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:15:40,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:15:40,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:15:40,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:40,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43566.37 MB 2025-02-15 07:15:40,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37441.45 MB 2025-02-15 07:15:40,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6124.92 MB 2025-02-15 07:15:40,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54079.26 MB 2025-02-15 07:15:40,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66790.10 MB 2025-02-15 07:15:40,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12710.84 MB 2025-02-15 07:15:40,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60554.88 MB 2025-02-15 07:15:42,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:15:42,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:15:42,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 07:15:42,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37441.45 MB 2025-02-15 07:15:42,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37972.29 MB 2025-02-15 07:15:42,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:15:42,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66790.10 MB 2025-02-15 07:15:42,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45348.81 MB 2025-02-15 07:15:42,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21441.28 MB 2025-02-15 07:15:42,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41950.83 MB 2025-02-15 07:15:42,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:15:42,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:15:42,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:15:42,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37972.29 MB 2025-02-15 07:15:42,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39861.82 MB 2025-02-15 07:15:42,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:15:42,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45348.81 MB 2025-02-15 07:15:42,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46292.53 MB 2025-02-15 07:15:42,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:15:42,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41279.25 MB 2025-02-15 07:15:42,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:15:42,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:15:42,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:15:42,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39861.82 MB 2025-02-15 07:15:42,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42103.68 MB 2025-02-15 07:15:42,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:15:42,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46292.53 MB 2025-02-15 07:15:42,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51954.84 MB 2025-02-15 07:15:42,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:15:42,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47647.96 MB 2025-02-15 07:15:42,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:15:42,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:15:42,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:15:42,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37972.29 MB 2025-02-15 07:15:42,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42103.68 MB 2025-02-15 07:15:42,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:15:42,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45348.81 MB 2025-02-15 07:15:42,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51954.84 MB 2025-02-15 07:15:42,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:15:42,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47647.96 MB 2025-02-15 07:15:42,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:15:42,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:15:42,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:15:42,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43637.22 MB 2025-02-15 07:15:42,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44404.22 MB 2025-02-15 07:15:42,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:15:42,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51954.84 MB 2025-02-15 07:15:42,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:15:42,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:15:42,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45112.01 MB 2025-02-15 07:15:42,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:15:42,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:15:42,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:15:42,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44817.11 MB 2025-02-15 07:15:42,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45045.01 MB 2025-02-15 07:15:42,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.90 MB 2025-02-15 07:15:42,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:15:42,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:15:42,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:15:42,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45261.99 MB 2025-02-15 07:15:42,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:15:42,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:15:42,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.57 seconds 2025-02-15 07:15:42,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:42,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31746.62 MB 2025-02-15 07:15:42,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45245.37 MB 2025-02-15 07:15:42,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13498.76 MB 2025-02-15 07:15:42,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52441.38 MB 2025-02-15 07:15:42,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:15:42,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -75.50 MB 2025-02-15 07:15:42,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45261.99 MB 2025-02-15 07:15:43,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:15:43,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:15:43,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:15:43,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:43,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45245.37 MB 2025-02-15 07:15:43,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36739.91 MB 2025-02-15 07:15:43,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8505.47 MB 2025-02-15 07:15:43,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:15:43,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52365.89 MB 2025-02-15 07:15:43,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:15:43,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47748.62 MB 2025-02-15 07:15:43,272 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 07:15:43,273 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:15:43,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:15:43,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:15:43,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:15:43,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:15:43,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36739.91 MB 2025-02-15 07:15:43,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45149.21 MB 2025-02-15 07:15:43,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 07:15:43,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52365.89 MB 2025-02-15 07:15:43,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60725.13 MB 2025-02-15 07:15:43,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 07:15:43,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45149.21 MB 2025-02-15 07:15:43,439 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 07:15:43,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:43,441 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:15:43,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:43,442 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:15:43,446 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:15:43,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:15:43,447 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:15:43,448 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:16:37,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:16:37,616 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:16:37,622 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:16:37,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:16:37,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1344, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:16:37,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:16:37,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1344, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:16:58,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:16:58,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:16:58,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.94 seconds 2025-02-15 07:16:58,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:16:58,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35248.12 MB 2025-02-15 07:16:58,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40004.46 MB 2025-02-15 07:16:58,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4756.34 MB 2025-02-15 07:16:58,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69084.38 MB 2025-02-15 07:16:58,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52852.42 MB 2025-02-15 07:16:58,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16231.96 MB 2025-02-15 07:16:58,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49022.84 MB 2025-02-15 07:16:58,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:16:58,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:16:58,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:16:58,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:16:58,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40004.46 MB 2025-02-15 07:16:58,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35679.09 MB 2025-02-15 07:16:58,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4325.37 MB 2025-02-15 07:16:58,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52852.42 MB 2025-02-15 07:16:58,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62216.21 MB 2025-02-15 07:16:58,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9363.78 MB 2025-02-15 07:16:58,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54084.51 MB 2025-02-15 07:17:00,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:17:00,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:17:00,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:17:00,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:00,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35679.09 MB 2025-02-15 07:17:00,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36209.93 MB 2025-02-15 07:17:00,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:17:00,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62216.21 MB 2025-02-15 07:17:00,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 07:17:00,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14120.12 MB 2025-02-15 07:17:00,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40188.48 MB 2025-02-15 07:17:00,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:17:00,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:17:00,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:17:00,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:00,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36209.93 MB 2025-02-15 07:17:00,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38099.46 MB 2025-02-15 07:17:00,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:17:00,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:00,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 07:17:00,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:00,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39516.89 MB 2025-02-15 07:17:00,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:17:00,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:17:00,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:17:00,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:00,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38099.46 MB 2025-02-15 07:17:00,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40341.32 MB 2025-02-15 07:17:00,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:17:00,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:00,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49983.52 MB 2025-02-15 07:17:00,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 07:17:00,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45885.60 MB 2025-02-15 07:17:00,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:17:00,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:17:00,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:17:00,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:00,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36209.93 MB 2025-02-15 07:17:00,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40341.32 MB 2025-02-15 07:17:00,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:17:00,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:00,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49983.52 MB 2025-02-15 07:17:00,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 07:17:00,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45885.60 MB 2025-02-15 07:17:01,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:17:01,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:17:01,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 07:17:01,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:01,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41874.86 MB 2025-02-15 07:17:01,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42641.86 MB 2025-02-15 07:17:01,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:17:01,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49983.52 MB 2025-02-15 07:17:01,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:17:01,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:17:01,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43349.65 MB 2025-02-15 07:17:01,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:17:01,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:17:01,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:17:01,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:01,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43054.75 MB 2025-02-15 07:17:01,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43282.98 MB 2025-02-15 07:17:01,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 07:17:01,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:17:01,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:17:01,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:01,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43524.06 MB 2025-02-15 07:17:01,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:17:01,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:17:01,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.39 seconds 2025-02-15 07:17:01,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:01,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30565.51 MB 2025-02-15 07:17:01,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43483.29 MB 2025-02-15 07:17:01,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12917.78 MB 2025-02-15 07:17:01,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69084.38 MB 2025-02-15 07:17:01,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:17:01,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18689.82 MB 2025-02-15 07:17:01,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43524.06 MB 2025-02-15 07:17:01,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:17:01,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:17:01,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:17:01,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:01,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43483.29 MB 2025-02-15 07:17:01,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35558.09 MB 2025-02-15 07:17:01,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7925.20 MB 2025-02-15 07:17:01,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:17:01,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:17:01,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:01,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45985.97 MB 2025-02-15 07:17:01,311 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 07:17:01,311 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:17:01,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:17:01,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:17:01,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:17:01,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:01,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35558.09 MB 2025-02-15 07:17:01,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43965.83 MB 2025-02-15 07:17:01,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 07:17:01,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:17:01,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58753.81 MB 2025-02-15 07:17:01,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 07:17:01,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43965.83 MB 2025-02-15 07:17:01,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 07:17:01,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:01,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:17:01,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:01,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:17:01,482 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:17:01,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:01,483 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:17:01,483 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:17:31,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:31,437 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:17:31,442 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:17:31,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:31,446 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1605, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:17:31,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:31,447 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1605, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:17:56,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:17:56,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:17:56,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.08 seconds 2025-02-15 07:17:56,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:56,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37066.81 MB 2025-02-15 07:17:56,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42746.81 MB 2025-02-15 07:17:56,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5680.01 MB 2025-02-15 07:17:56,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67113.06 MB 2025-02-15 07:17:56,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53777.27 MB 2025-02-15 07:17:56,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13335.79 MB 2025-02-15 07:17:56,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51747.50 MB 2025-02-15 07:17:56,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:17:56,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:17:56,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:17:56,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:56,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42746.81 MB 2025-02-15 07:17:56,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37035.95 MB 2025-02-15 07:17:56,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5710.86 MB 2025-02-15 07:17:56,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53777.27 MB 2025-02-15 07:17:56,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64453.87 MB 2025-02-15 07:17:56,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10676.60 MB 2025-02-15 07:17:56,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58802.66 MB 2025-02-15 07:17:58,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:17:58,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:17:58,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 07:17:58,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:58,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37035.95 MB 2025-02-15 07:17:58,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37566.79 MB 2025-02-15 07:17:58,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:17:58,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64453.87 MB 2025-02-15 07:17:58,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 07:17:58,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16357.79 MB 2025-02-15 07:17:58,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41545.34 MB 2025-02-15 07:17:58,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:17:58,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:17:58,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:17:58,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:58,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37566.79 MB 2025-02-15 07:17:58,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39456.32 MB 2025-02-15 07:17:58,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:17:58,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:58,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48096.08 MB 2025-02-15 07:17:58,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:58,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40873.75 MB 2025-02-15 07:17:58,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:17:58,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:17:58,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:17:58,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:58,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39456.32 MB 2025-02-15 07:17:58,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41698.18 MB 2025-02-15 07:17:58,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:17:58,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:58,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50927.24 MB 2025-02-15 07:17:58,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:17:58,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47242.46 MB 2025-02-15 07:17:58,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:17:58,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:17:58,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:17:58,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:58,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37566.79 MB 2025-02-15 07:17:58,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41698.18 MB 2025-02-15 07:17:58,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:17:58,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-15 07:17:58,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50927.24 MB 2025-02-15 07:17:58,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:17:58,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47242.46 MB 2025-02-15 07:17:59,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:17:59,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:17:59,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 07:17:59,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:59,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43231.72 MB 2025-02-15 07:17:59,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43998.72 MB 2025-02-15 07:17:59,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:17:59,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50927.24 MB 2025-02-15 07:17:59,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51338.28 MB 2025-02-15 07:17:59,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:17:59,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44706.51 MB 2025-02-15 07:17:59,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:17:59,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:17:59,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 07:17:59,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:59,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44411.61 MB 2025-02-15 07:17:59,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44638.26 MB 2025-02-15 07:17:59,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.65 MB 2025-02-15 07:17:59,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51338.28 MB 2025-02-15 07:17:59,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51338.28 MB 2025-02-15 07:17:59,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:59,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44869.26 MB 2025-02-15 07:17:59,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:17:59,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:17:59,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.68 seconds 2025-02-15 07:17:59,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:59,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31474.86 MB 2025-02-15 07:17:59,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44839.11 MB 2025-02-15 07:17:59,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13364.26 MB 2025-02-15 07:17:59,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67113.06 MB 2025-02-15 07:17:59,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51338.28 MB 2025-02-15 07:17:59,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15774.78 MB 2025-02-15 07:17:59,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44869.26 MB 2025-02-15 07:17:59,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:17:59,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:17:59,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 07:17:59,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:59,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44839.11 MB 2025-02-15 07:17:59,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36463.73 MB 2025-02-15 07:17:59,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8375.38 MB 2025-02-15 07:17:59,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51338.28 MB 2025-02-15 07:17:59,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51338.28 MB 2025-02-15 07:17:59,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:17:59,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47337.88 MB 2025-02-15 07:17:59,441 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 07:17:59,441 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:17:59,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:17:59,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:17:59,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:17:59,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:17:59,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36463.73 MB 2025-02-15 07:17:59,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44860.38 MB 2025-02-15 07:17:59,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-15 07:17:59,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51338.28 MB 2025-02-15 07:17:59,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59684.95 MB 2025-02-15 07:17:59,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 07:17:59,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44860.38 MB 2025-02-15 07:17:59,675 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 07:17:59,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:59,677 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:17:59,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:59,678 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:17:59,683 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:17:59,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:17:59,684 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:17:59,684 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:19:23,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:23,111 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:19:23,116 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:19:23,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:23,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:19:23,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:23,122 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:19:33,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:19:33,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:19:33,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.04 seconds 2025-02-15 07:19:33,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:33,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30412.21 MB 2025-02-15 07:19:33,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32712.53 MB 2025-02-15 07:19:33,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2300.31 MB 2025-02-15 07:19:33,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68031.61 MB 2025-02-15 07:19:33,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39736.84 MB 2025-02-15 07:19:33,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28294.77 MB 2025-02-15 07:19:33,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41695.52 MB 2025-02-15 07:19:33,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:19:33,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:19:33,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 07:19:33,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:33,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32712.53 MB 2025-02-15 07:19:33,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32072.25 MB 2025-02-15 07:19:33,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -640.28 MB 2025-02-15 07:19:33,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39736.84 MB 2025-02-15 07:19:33,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45199.92 MB 2025-02-15 07:19:33,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5463.08 MB 2025-02-15 07:19:33,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41398.90 MB 2025-02-15 07:19:35,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:19:35,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:19:35,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 07:19:35,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32072.25 MB 2025-02-15 07:19:35,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32603.09 MB 2025-02-15 07:19:35,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:19:35,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45199.92 MB 2025-02-15 07:19:35,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41861.25 MB 2025-02-15 07:19:35,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3338.67 MB 2025-02-15 07:19:35,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36581.64 MB 2025-02-15 07:19:35,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:19:35,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:19:35,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:19:35,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32603.09 MB 2025-02-15 07:19:35,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34492.62 MB 2025-02-15 07:19:35,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:19:35,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41861.25 MB 2025-02-15 07:19:35,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41861.25 MB 2025-02-15 07:19:35,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:19:35,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35910.05 MB 2025-02-15 07:19:35,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:19:35,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:19:35,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:19:35,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34492.62 MB 2025-02-15 07:19:35,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36734.48 MB 2025-02-15 07:19:35,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:19:35,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41861.25 MB 2025-02-15 07:19:35,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45636.12 MB 2025-02-15 07:19:35,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 07:19:35,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42278.76 MB 2025-02-15 07:19:35,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:19:35,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:19:35,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 07:19:35,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32603.09 MB 2025-02-15 07:19:35,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36734.48 MB 2025-02-15 07:19:35,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:19:35,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41861.25 MB 2025-02-15 07:19:35,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45636.12 MB 2025-02-15 07:19:35,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 07:19:35,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42278.76 MB 2025-02-15 07:19:35,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:19:35,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:19:35,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:19:35,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38268.02 MB 2025-02-15 07:19:35,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39035.02 MB 2025-02-15 07:19:35,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:19:35,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45636.12 MB 2025-02-15 07:19:35,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46047.17 MB 2025-02-15 07:19:35,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:19:35,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39742.81 MB 2025-02-15 07:19:35,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:19:35,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:19:35,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:19:35,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39447.91 MB 2025-02-15 07:19:35,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39676.99 MB 2025-02-15 07:19:35,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.07 MB 2025-02-15 07:19:35,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46047.17 MB 2025-02-15 07:19:35,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46047.17 MB 2025-02-15 07:19:35,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:19:35,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39888.16 MB 2025-02-15 07:19:35,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:19:35,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:19:35,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.42 seconds 2025-02-15 07:19:35,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28147.56 MB 2025-02-15 07:19:35,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39878.06 MB 2025-02-15 07:19:35,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11730.50 MB 2025-02-15 07:19:35,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68031.61 MB 2025-02-15 07:19:35,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46047.17 MB 2025-02-15 07:19:35,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21984.44 MB 2025-02-15 07:19:35,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39888.16 MB 2025-02-15 07:19:35,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:19:35,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:19:35,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:19:35,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39878.06 MB 2025-02-15 07:19:35,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33151.41 MB 2025-02-15 07:19:35,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6726.65 MB 2025-02-15 07:19:35,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46047.17 MB 2025-02-15 07:19:35,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46047.17 MB 2025-02-15 07:19:35,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:19:35,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42389.73 MB 2025-02-15 07:19:35,836 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:19:35,836 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:19:35,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:19:35,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:19:35,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:19:35,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:19:35,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33151.41 MB 2025-02-15 07:19:35,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41590.43 MB 2025-02-15 07:19:35,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:19:35,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46047.17 MB 2025-02-15 07:19:35,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54437.87 MB 2025-02-15 07:19:35,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:19:35,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41590.43 MB 2025-02-15 07:19:36,004 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:19:36,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:36,006 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:19:36,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:36,007 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:19:36,012 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:19:36,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:36,013 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:19:36,013 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:19:53,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:53,814 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:19:53,819 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:19:53,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:53,822 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1710, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:19:53,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:19:53,823 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1710, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:20:20,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:20:20,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:20:20,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.60 seconds 2025-02-15 07:20:20,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:20,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37798.46 MB 2025-02-15 07:20:20,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43850.84 MB 2025-02-15 07:20:20,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6052.38 MB 2025-02-15 07:20:20,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67022.88 MB 2025-02-15 07:20:20,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54175.73 MB 2025-02-15 07:20:20,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12847.15 MB 2025-02-15 07:20:20,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52705.65 MB 2025-02-15 07:20:20,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:20:20,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:20:20,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:20:20,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:20,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43850.84 MB 2025-02-15 07:20:20,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37581.81 MB 2025-02-15 07:20:20,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6269.03 MB 2025-02-15 07:20:20,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54175.73 MB 2025-02-15 07:20:20,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67408.76 MB 2025-02-15 07:20:20,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13233.03 MB 2025-02-15 07:20:20,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61616.91 MB 2025-02-15 07:20:22,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:20:22,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:20:22,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:20:22,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37581.81 MB 2025-02-15 07:20:22,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38112.65 MB 2025-02-15 07:20:22,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:20:22,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67408.76 MB 2025-02-15 07:20:22,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45359.30 MB 2025-02-15 07:20:22,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22049.46 MB 2025-02-15 07:20:22,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42091.20 MB 2025-02-15 07:20:22,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:20:22,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:20:22,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:20:22,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38112.65 MB 2025-02-15 07:20:22,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40002.18 MB 2025-02-15 07:20:22,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:20:22,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45359.30 MB 2025-02-15 07:20:22,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46303.02 MB 2025-02-15 07:20:22,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:20:22,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41419.61 MB 2025-02-15 07:20:22,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:20:22,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:20:22,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:20:22,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40002.18 MB 2025-02-15 07:20:22,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42244.04 MB 2025-02-15 07:20:22,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:20:22,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46303.02 MB 2025-02-15 07:20:22,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51965.33 MB 2025-02-15 07:20:22,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:20:22,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47788.32 MB 2025-02-15 07:20:22,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:20:22,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:20:22,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:20:22,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38112.65 MB 2025-02-15 07:20:22,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42244.04 MB 2025-02-15 07:20:22,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:20:22,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45359.30 MB 2025-02-15 07:20:22,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51965.33 MB 2025-02-15 07:20:22,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:20:22,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47788.32 MB 2025-02-15 07:20:22,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:20:22,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:20:22,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:20:22,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43777.58 MB 2025-02-15 07:20:22,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44544.58 MB 2025-02-15 07:20:22,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:20:22,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51965.33 MB 2025-02-15 07:20:22,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:20:22,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:20:22,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45252.37 MB 2025-02-15 07:20:22,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:20:22,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:20:22,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:20:22,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44957.47 MB 2025-02-15 07:20:22,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45185.38 MB 2025-02-15 07:20:22,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.90 MB 2025-02-15 07:20:22,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:20:22,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:20:22,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:20:22,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45408.51 MB 2025-02-15 07:20:22,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:20:22,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:20:22,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.05 seconds 2025-02-15 07:20:22,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:22,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31840.69 MB 2025-02-15 07:20:22,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45386.23 MB 2025-02-15 07:20:22,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13545.54 MB 2025-02-15 07:20:22,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67022.88 MB 2025-02-15 07:20:22,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:20:22,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14644.41 MB 2025-02-15 07:20:22,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45408.51 MB 2025-02-15 07:20:23,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:20:23,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:20:23,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:20:23,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:23,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45386.23 MB 2025-02-15 07:20:23,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36833.98 MB 2025-02-15 07:20:23,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8552.25 MB 2025-02-15 07:20:23,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:20:23,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:20:23,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:20:23,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47888.99 MB 2025-02-15 07:20:23,160 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 07:20:23,161 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:20:23,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:20:23,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:20:23,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:20:23,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:20:23,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36833.98 MB 2025-02-15 07:20:23,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45243.28 MB 2025-02-15 07:20:23,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 07:20:23,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:20:23,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60737.72 MB 2025-02-15 07:20:23,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 07:20:23,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45243.28 MB 2025-02-15 07:20:23,324 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 07:20:23,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:20:23,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:20:23,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:20:23,327 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:20:23,332 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:20:23,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:20:23,333 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:20:23,333 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:21:40,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:40,208 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:21:40,213 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:21:40,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:40,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 350, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:21:40,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:40,218 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 350, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:21:45,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:21:45,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:21:45,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.41 seconds 2025-02-15 07:21:45,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:45,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28321.76 MB 2025-02-15 07:21:45,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29560.40 MB 2025-02-15 07:21:45,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1238.63 MB 2025-02-15 07:21:45,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69096.96 MB 2025-02-15 07:21:45,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-15 07:21:45,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32593.94 MB 2025-02-15 07:21:45,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38472.61 MB 2025-02-15 07:21:45,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:21:45,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:21:45,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:21:45,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:45,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29560.40 MB 2025-02-15 07:21:45,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30034.03 MB 2025-02-15 07:21:45,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 473.63 MB 2025-02-15 07:21:45,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-15 07:21:45,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37679.53 MB 2025-02-15 07:21:45,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1176.50 MB 2025-02-15 07:21:45,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34223.68 MB 2025-02-15 07:21:47,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:21:47,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:21:47,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-15 07:21:47,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30034.03 MB 2025-02-15 07:21:47,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30474.63 MB 2025-02-15 07:21:47,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-15 07:21:47,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37679.53 MB 2025-02-15 07:21:47,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37679.53 MB 2025-02-15 07:21:47,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:21:47,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34458.48 MB 2025-02-15 07:21:47,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:21:47,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:21:47,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:21:47,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30474.63 MB 2025-02-15 07:21:47,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32042.76 MB 2025-02-15 07:21:47,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1568.13 MB 2025-02-15 07:21:47,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37679.53 MB 2025-02-15 07:21:47,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37679.53 MB 2025-02-15 07:21:47,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:21:47,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33219.22 MB 2025-02-15 07:21:47,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:21:47,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:21:47,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 07:21:47,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32042.76 MB 2025-02-15 07:21:47,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33904.56 MB 2025-02-15 07:21:47,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1861.80 MB 2025-02-15 07:21:47,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37679.53 MB 2025-02-15 07:21:47,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42385.54 MB 2025-02-15 07:21:47,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4706.01 MB 2025-02-15 07:21:47,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38510.49 MB 2025-02-15 07:21:47,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:21:47,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:21:47,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 07:21:47,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30474.63 MB 2025-02-15 07:21:47,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33904.56 MB 2025-02-15 07:21:47,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3429.93 MB 2025-02-15 07:21:47,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37679.53 MB 2025-02-15 07:21:47,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42385.54 MB 2025-02-15 07:21:47,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4706.01 MB 2025-02-15 07:21:47,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38510.49 MB 2025-02-15 07:21:47,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:21:47,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:21:47,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:21:47,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35177.40 MB 2025-02-15 07:21:47,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35814.01 MB 2025-02-15 07:21:47,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 636.61 MB 2025-02-15 07:21:47,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42385.54 MB 2025-02-15 07:21:47,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42723.18 MB 2025-02-15 07:21:47,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 337.64 MB 2025-02-15 07:21:47,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36401.47 MB 2025-02-15 07:21:47,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:21:47,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:21:47,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:21:47,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36156.71 MB 2025-02-15 07:21:47,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36370.57 MB 2025-02-15 07:21:47,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.86 MB 2025-02-15 07:21:47,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42723.18 MB 2025-02-15 07:21:47,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42725.28 MB 2025-02-15 07:21:47,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 07:21:47,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36536.07 MB 2025-02-15 07:21:47,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:21:47,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:21:47,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.40 seconds 2025-02-15 07:21:47,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27102.34 MB 2025-02-15 07:21:47,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36571.64 MB 2025-02-15 07:21:47,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9469.30 MB 2025-02-15 07:21:47,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69096.96 MB 2025-02-15 07:21:47,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42725.28 MB 2025-02-15 07:21:47,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26371.69 MB 2025-02-15 07:21:47,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36571.64 MB 2025-02-15 07:21:47,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:21:47,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:21:47,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:21:47,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36571.64 MB 2025-02-15 07:21:47,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39585.67 MB 2025-02-15 07:21:47,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 07:21:47,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42725.28 MB 2025-02-15 07:21:47,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42725.28 MB 2025-02-15 07:21:47,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:21:47,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39887.04 MB 2025-02-15 07:21:47,905 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:21:47,905 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:21:47,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:21:47,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:21:47,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:21:47,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:21:47,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31785.29 MB 2025-02-15 07:21:47,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40224.31 MB 2025-02-15 07:21:47,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:21:47,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42725.28 MB 2025-02-15 07:21:47,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53215.23 MB 2025-02-15 07:21:47,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 07:21:47,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40224.31 MB 2025-02-15 07:21:48,072 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:21:48,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:48,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:21:48,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:48,075 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:21:48,080 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:21:48,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:21:48,081 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:21:48,081 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:22:45,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:22:45,047 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:22:45,052 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:22:45,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:22:45,056 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1832, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:22:45,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:22:45,057 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1832, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:23:13,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:23:13,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:23:13,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.37 seconds 2025-02-15 07:23:13,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:13,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38648.58 MB 2025-02-15 07:23:13,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45132.97 MB 2025-02-15 07:23:13,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6484.39 MB 2025-02-15 07:23:13,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65800.24 MB 2025-02-15 07:23:13,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54624.52 MB 2025-02-15 07:23:13,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11175.72 MB 2025-02-15 07:23:13,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54008.75 MB 2025-02-15 07:23:13,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:23:13,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:23:13,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 07:23:13,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:13,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45132.97 MB 2025-02-15 07:23:13,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38216.05 MB 2025-02-15 07:23:13,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6916.92 MB 2025-02-15 07:23:13,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54624.52 MB 2025-02-15 07:23:13,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68474.11 MB 2025-02-15 07:23:13,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13849.59 MB 2025-02-15 07:23:13,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63831.51 MB 2025-02-15 07:23:15,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:23:15,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:23:15,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:23:15,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38216.05 MB 2025-02-15 07:23:15,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38746.89 MB 2025-02-15 07:23:15,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:23:15,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68474.11 MB 2025-02-15 07:23:15,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-15 07:23:15,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23112.71 MB 2025-02-15 07:23:15,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42725.44 MB 2025-02-15 07:23:15,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:23:15,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:23:15,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:23:15,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38746.89 MB 2025-02-15 07:23:15,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40636.42 MB 2025-02-15 07:23:15,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:23:15,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 07:23:15,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46305.12 MB 2025-02-15 07:23:15,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:23:15,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42053.85 MB 2025-02-15 07:23:15,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:23:15,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:23:15,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:23:15,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40636.42 MB 2025-02-15 07:23:15,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42878.28 MB 2025-02-15 07:23:15,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:23:15,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46305.12 MB 2025-02-15 07:23:15,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51967.43 MB 2025-02-15 07:23:15,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:23:15,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48422.56 MB 2025-02-15 07:23:15,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:23:15,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:23:15,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:23:15,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38746.89 MB 2025-02-15 07:23:15,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42878.28 MB 2025-02-15 07:23:15,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:23:15,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 07:23:15,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51967.43 MB 2025-02-15 07:23:15,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:23:15,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48422.56 MB 2025-02-15 07:23:15,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:23:15,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:23:15,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:23:15,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44411.82 MB 2025-02-15 07:23:15,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45178.82 MB 2025-02-15 07:23:15,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:23:15,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51967.43 MB 2025-02-15 07:23:15,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:23:15,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:23:15,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45886.61 MB 2025-02-15 07:23:15,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:23:15,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:23:15,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:23:15,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45591.71 MB 2025-02-15 07:23:15,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45819.54 MB 2025-02-15 07:23:15,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.83 MB 2025-02-15 07:23:15,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:23:15,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:23:15,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:23:15,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46034.36 MB 2025-02-15 07:23:15,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:23:15,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:23:15,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.87 seconds 2025-02-15 07:23:15,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:15,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32265.74 MB 2025-02-15 07:23:15,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46019.83 MB 2025-02-15 07:23:15,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13754.09 MB 2025-02-15 07:23:15,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65800.24 MB 2025-02-15 07:23:15,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:23:15,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13421.77 MB 2025-02-15 07:23:15,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46034.36 MB 2025-02-15 07:23:16,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:23:16,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:23:16,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:23:16,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:16,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46019.83 MB 2025-02-15 07:23:16,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37257.97 MB 2025-02-15 07:23:16,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8761.86 MB 2025-02-15 07:23:16,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:23:16,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52378.47 MB 2025-02-15 07:23:16,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:23:16,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48522.23 MB 2025-02-15 07:23:16,242 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 07:23:16,243 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:23:16,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:23:16,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:23:16,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:23:16,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:16,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37257.97 MB 2025-02-15 07:23:16,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45663.63 MB 2025-02-15 07:23:16,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 07:23:16,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-15 07:23:16,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56558.09 MB 2025-02-15 07:23:16,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 07:23:16,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45663.63 MB 2025-02-15 07:23:16,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 07:23:16,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:16,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:23:16,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:16,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:23:16,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:23:16,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:16,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:23:16,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:23:25,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:25,068 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:23:25,074 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:23:25,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:25,078 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1322, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:23:25,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:25,079 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1322, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:23:45,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:23:45,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:23:45,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.69 seconds 2025-02-15 07:23:45,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:45,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35094.82 MB 2025-02-15 07:23:45,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39773.56 MB 2025-02-15 07:23:45,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4678.75 MB 2025-02-15 07:23:45,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64917.34 MB 2025-02-15 07:23:45,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52776.93 MB 2025-02-15 07:23:45,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12140.41 MB 2025-02-15 07:23:45,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48643.05 MB 2025-02-15 07:23:45,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:23:45,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:23:45,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:23:45,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:45,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39773.56 MB 2025-02-15 07:23:45,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35564.72 MB 2025-02-15 07:23:45,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4208.84 MB 2025-02-15 07:23:45,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52776.93 MB 2025-02-15 07:23:45,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61972.94 MB 2025-02-15 07:23:45,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9196.01 MB 2025-02-15 07:23:45,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53599.35 MB 2025-02-15 07:23:47,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:23:47,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:23:47,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:23:47,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:47,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35564.72 MB 2025-02-15 07:23:47,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36095.56 MB 2025-02-15 07:23:47,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:23:47,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61972.94 MB 2025-02-15 07:23:47,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48098.18 MB 2025-02-15 07:23:47,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13874.76 MB 2025-02-15 07:23:47,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40074.11 MB 2025-02-15 07:23:47,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:23:47,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:23:47,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:23:47,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:47,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36095.56 MB 2025-02-15 07:23:47,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37985.09 MB 2025-02-15 07:23:47,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:23:47,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48098.18 MB 2025-02-15 07:23:47,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48098.18 MB 2025-02-15 07:23:47,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:23:47,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39402.52 MB 2025-02-15 07:23:48,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:23:48,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:23:48,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:23:48,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37985.09 MB 2025-02-15 07:23:48,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40226.95 MB 2025-02-15 07:23:48,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:23:48,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48098.18 MB 2025-02-15 07:23:48,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49985.62 MB 2025-02-15 07:23:48,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 07:23:48,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45771.23 MB 2025-02-15 07:23:48,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:23:48,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:23:48,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:23:48,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36095.56 MB 2025-02-15 07:23:48,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40226.95 MB 2025-02-15 07:23:48,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:23:48,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48098.18 MB 2025-02-15 07:23:48,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49985.62 MB 2025-02-15 07:23:48,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 07:23:48,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45771.23 MB 2025-02-15 07:23:48,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:23:48,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:23:48,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:23:48,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41760.49 MB 2025-02-15 07:23:48,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42527.49 MB 2025-02-15 07:23:48,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:23:48,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49985.62 MB 2025-02-15 07:23:48,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:23:48,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-15 07:23:48,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43235.28 MB 2025-02-15 07:23:48,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:23:48,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:23:48,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:23:48,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42940.38 MB 2025-02-15 07:23:48,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43169.12 MB 2025-02-15 07:23:48,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-15 07:23:48,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:23:48,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:23:48,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:23:48,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43397.88 MB 2025-02-15 07:23:48,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:23:48,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:23:48,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.12 seconds 2025-02-15 07:23:48,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30488.86 MB 2025-02-15 07:23:48,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43369.75 MB 2025-02-15 07:23:48,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12880.89 MB 2025-02-15 07:23:48,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64917.34 MB 2025-02-15 07:23:48,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:23:48,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14522.78 MB 2025-02-15 07:23:48,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43397.88 MB 2025-02-15 07:23:48,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:23:48,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:23:48,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:23:48,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43369.75 MB 2025-02-15 07:23:48,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35486.08 MB 2025-02-15 07:23:48,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7883.68 MB 2025-02-15 07:23:48,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:23:48,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50394.56 MB 2025-02-15 07:23:48,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:23:48,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45876.11 MB 2025-02-15 07:23:48,493 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 07:23:48,493 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:23:48,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:23:48,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:23:48,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:23:48,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:23:48,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35486.08 MB 2025-02-15 07:23:48,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43906.85 MB 2025-02-15 07:23:48,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 07:23:48,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50394.56 MB 2025-02-15 07:23:48,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58766.39 MB 2025-02-15 07:23:48,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 07:23:48,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43906.85 MB 2025-02-15 07:23:48,660 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 07:23:48,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:48,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:23:48,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:48,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:23:48,667 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:23:48,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:23:48,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:23:48,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:24:34,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:34,234 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:24:34,241 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:24:34,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:34,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 172, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:24:34,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:34,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 172, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:24:37,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:24:37,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:24:37,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.86 seconds 2025-02-15 07:24:37,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:37,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27081.43 MB 2025-02-15 07:24:37,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27690.13 MB 2025-02-15 07:24:37,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.70 MB 2025-02-15 07:24:37,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67138.22 MB 2025-02-15 07:24:37,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:37,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31581.01 MB 2025-02-15 07:24:37,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36552.80 MB 2025-02-15 07:24:37,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:24:37,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:24:37,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:24:37,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:37,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27690.13 MB 2025-02-15 07:24:37,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27907.79 MB 2025-02-15 07:24:37,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.66 MB 2025-02-15 07:24:37,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:37,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:37,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:37,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29951.61 MB 2025-02-15 07:24:37,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:24:37,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:24:37,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 07:24:37,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:37,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27907.79 MB 2025-02-15 07:24:37,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28121.45 MB 2025-02-15 07:24:37,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 07:24:37,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:37,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:37,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:37,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32077.44 MB 2025-02-15 07:24:37,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:24:37,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:24:37,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:24:37,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:37,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28121.39 MB 2025-02-15 07:24:37,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28881.74 MB 2025-02-15 07:24:37,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 07:24:37,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:37,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:37,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:37,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29452.26 MB 2025-02-15 07:24:38,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:24:38,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:24:38,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:24:38,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28881.74 MB 2025-02-15 07:24:38,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29784.13 MB 2025-02-15 07:24:38,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 07:24:38,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:38,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:38,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:38,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32015.66 MB 2025-02-15 07:24:38,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:24:38,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:24:38,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 07:24:38,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28121.39 MB 2025-02-15 07:24:38,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29784.13 MB 2025-02-15 07:24:38,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 07:24:38,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:38,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-15 07:24:38,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:38,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32015.66 MB 2025-02-15 07:24:38,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:24:38,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:24:38,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:24:38,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30401.38 MB 2025-02-15 07:24:38,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30710.10 MB 2025-02-15 07:24:38,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 07:24:38,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-15 07:24:38,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:24:38,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-15 07:24:38,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31002.62 MB 2025-02-15 07:24:38,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:24:38,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:24:38,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:24:38,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30876.29 MB 2025-02-15 07:24:38,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31104.60 MB 2025-02-15 07:24:38,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-15 07:24:38,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:24:38,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:24:38,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:38,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31118.44 MB 2025-02-15 07:24:38,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:24:38,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:24:38,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.93 seconds 2025-02-15 07:24:38,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26482.17 MB 2025-02-15 07:24:38,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31305.57 MB 2025-02-15 07:24:38,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4823.40 MB 2025-02-15 07:24:38,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67138.22 MB 2025-02-15 07:24:38,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:24:38,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31419.53 MB 2025-02-15 07:24:38,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31305.57 MB 2025-02-15 07:24:38,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:24:38,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:24:38,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 07:24:38,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31305.57 MB 2025-02-15 07:24:38,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30356.58 MB 2025-02-15 07:24:38,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -948.99 MB 2025-02-15 07:24:38,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:24:38,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:24:38,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:24:38,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32108.91 MB 2025-02-15 07:24:38,494 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 07:24:38,494 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:24:38,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:24:38,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:24:38,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:24:38,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:24:38,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30356.58 MB 2025-02-15 07:24:38,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38791.43 MB 2025-02-15 07:24:38,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 07:24:38,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:24:38,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44105.20 MB 2025-02-15 07:24:38,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 07:24:38,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38791.43 MB 2025-02-15 07:24:38,752 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 07:24:38,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:38,754 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:24:38,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:38,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:24:38,764 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:24:38,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:38,766 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:24:38,766 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:24:48,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:48,204 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:24:48,209 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:24:48,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:48,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1069, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:24:48,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:24:48,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1069, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:25:04,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:25:04,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:25:04,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.76 seconds 2025-02-15 07:25:04,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:04,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33331.87 MB 2025-02-15 07:25:04,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37115.13 MB 2025-02-15 07:25:04,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3783.26 MB 2025-02-15 07:25:04,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56683.92 MB 2025-02-15 07:25:04,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43530.58 MB 2025-02-15 07:25:04,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13153.34 MB 2025-02-15 07:25:04,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45974.14 MB 2025-02-15 07:25:05,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:25:05,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:25:05,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:25:05,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:05,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37115.13 MB 2025-02-15 07:25:05,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34249.45 MB 2025-02-15 07:25:05,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2865.68 MB 2025-02-15 07:25:05,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43530.58 MB 2025-02-15 07:25:05,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52435.09 MB 2025-02-15 07:25:05,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8904.51 MB 2025-02-15 07:25:05,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48827.67 MB 2025-02-15 07:25:06,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:25:06,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:25:06,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:25:06,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:06,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34249.45 MB 2025-02-15 07:25:06,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34780.29 MB 2025-02-15 07:25:06,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:25:06,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52435.09 MB 2025-02-15 07:25:06,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41162.90 MB 2025-02-15 07:25:06,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11272.19 MB 2025-02-15 07:25:06,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38758.84 MB 2025-02-15 07:25:07,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:25:07,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:25:07,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:25:07,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34780.29 MB 2025-02-15 07:25:07,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36669.83 MB 2025-02-15 07:25:07,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:25:07,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41162.90 MB 2025-02-15 07:25:07,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42106.62 MB 2025-02-15 07:25:07,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:25:07,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38087.25 MB 2025-02-15 07:25:07,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:25:07,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:25:07,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:25:07,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36669.83 MB 2025-02-15 07:25:07,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38911.68 MB 2025-02-15 07:25:07,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:25:07,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42106.62 MB 2025-02-15 07:25:07,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47768.93 MB 2025-02-15 07:25:07,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:25:07,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44455.96 MB 2025-02-15 07:25:07,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:25:07,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:25:07,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:25:07,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34780.29 MB 2025-02-15 07:25:07,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38911.68 MB 2025-02-15 07:25:07,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:25:07,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41162.90 MB 2025-02-15 07:25:07,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47768.93 MB 2025-02-15 07:25:07,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:25:07,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44455.96 MB 2025-02-15 07:25:07,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:25:07,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:25:07,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:25:07,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40445.22 MB 2025-02-15 07:25:07,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41212.23 MB 2025-02-15 07:25:07,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:25:07,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47768.93 MB 2025-02-15 07:25:07,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 07:25:07,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:25:07,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41920.01 MB 2025-02-15 07:25:07,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:25:07,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:25:07,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:25:07,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41625.11 MB 2025-02-15 07:25:07,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41854.20 MB 2025-02-15 07:25:07,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-15 07:25:07,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 07:25:07,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 07:25:07,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:07,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42095.11 MB 2025-02-15 07:25:07,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:25:07,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:25:07,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.19 seconds 2025-02-15 07:25:07,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29607.39 MB 2025-02-15 07:25:07,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42055.05 MB 2025-02-15 07:25:07,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12447.66 MB 2025-02-15 07:25:07,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56683.92 MB 2025-02-15 07:25:07,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 07:25:07,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8501.85 MB 2025-02-15 07:25:07,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42095.11 MB 2025-02-15 07:25:07,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:25:07,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:25:07,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:25:07,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42055.05 MB 2025-02-15 07:25:07,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34604.96 MB 2025-02-15 07:25:07,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7450.09 MB 2025-02-15 07:25:07,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 07:25:07,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-15 07:25:07,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:07,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44561.49 MB 2025-02-15 07:25:07,755 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 07:25:07,756 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:25:07,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:25:07,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:25:07,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:25:07,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:07,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34604.96 MB 2025-02-15 07:25:07,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43026.92 MB 2025-02-15 07:25:07,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-15 07:25:07,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48182.07 MB 2025-02-15 07:25:07,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 07:25:07,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 07:25:07,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43026.92 MB 2025-02-15 07:25:07,924 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 07:25:07,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:07,925 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:25:07,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:07,926 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:25:07,931 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:25:07,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:07,932 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:25:07,932 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:25:43,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:43,707 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:25:43,712 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:25:43,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:43,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:25:43,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:43,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:25:46,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:25:46,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:25:46,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-15 07:25:46,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:46,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.27 MB 2025-02-15 07:25:46,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27742.67 MB 2025-02-15 07:25:46,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-15 07:25:46,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67018.69 MB 2025-02-15 07:25:46,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:46,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31459.38 MB 2025-02-15 07:25:46,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36587.64 MB 2025-02-15 07:25:46,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:25:46,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:25:46,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:25:46,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:46,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27742.67 MB 2025-02-15 07:25:46,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27940.81 MB 2025-02-15 07:25:46,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.14 MB 2025-02-15 07:25:46,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:46,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:46,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:46,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30018.19 MB 2025-02-15 07:25:47,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:25:47,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:25:47,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 07:25:47,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27940.81 MB 2025-02-15 07:25:47,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28155.80 MB 2025-02-15 07:25:47,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-15 07:25:47,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:47,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:47,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:47,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32110.46 MB 2025-02-15 07:25:47,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:25:47,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:25:47,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:25:47,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28155.73 MB 2025-02-15 07:25:47,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28920.81 MB 2025-02-15 07:25:47,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-15 07:25:47,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:47,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:47,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:47,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29494.87 MB 2025-02-15 07:25:47,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:25:47,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:25:47,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:25:47,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28920.81 MB 2025-02-15 07:25:47,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29828.80 MB 2025-02-15 07:25:47,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-15 07:25:47,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:47,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:47,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:47,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32074.19 MB 2025-02-15 07:25:47,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:25:47,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:25:47,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:25:47,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28155.73 MB 2025-02-15 07:25:47,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29828.80 MB 2025-02-15 07:25:47,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-15 07:25:47,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:47,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 07:25:47,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:47,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32074.19 MB 2025-02-15 07:25:47,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:25:47,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:25:47,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:25:47,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30449.88 MB 2025-02-15 07:25:47,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30761.42 MB 2025-02-15 07:25:47,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.54 MB 2025-02-15 07:25:47,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 07:25:47,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35718.69 MB 2025-02-15 07:25:47,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-15 07:25:47,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31056.53 MB 2025-02-15 07:25:47,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:25:47,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:25:47,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:25:47,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30928.65 MB 2025-02-15 07:25:47,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31132.19 MB 2025-02-15 07:25:47,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.54 MB 2025-02-15 07:25:47,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35718.69 MB 2025-02-15 07:25:47,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35722.89 MB 2025-02-15 07:25:47,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 07:25:47,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31150.52 MB 2025-02-15 07:25:47,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:25:47,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:25:47,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.77 seconds 2025-02-15 07:25:47,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26499.59 MB 2025-02-15 07:25:47,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31332.89 MB 2025-02-15 07:25:47,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4833.30 MB 2025-02-15 07:25:47,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67018.69 MB 2025-02-15 07:25:47,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35722.89 MB 2025-02-15 07:25:47,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31295.80 MB 2025-02-15 07:25:47,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31332.89 MB 2025-02-15 07:25:47,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:25:47,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:25:47,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:25:47,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31332.89 MB 2025-02-15 07:25:47,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30375.43 MB 2025-02-15 07:25:47,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -957.46 MB 2025-02-15 07:25:47,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35722.89 MB 2025-02-15 07:25:47,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35722.89 MB 2025-02-15 07:25:47,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:25:47,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32034.87 MB 2025-02-15 07:25:47,774 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 07:25:47,775 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:25:47,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:25:47,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:25:47,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:25:47,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:25:47,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30375.43 MB 2025-02-15 07:25:47,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38798.64 MB 2025-02-15 07:25:47,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 07:25:47,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35722.89 MB 2025-02-15 07:25:47,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44098.91 MB 2025-02-15 07:25:47,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 07:25:47,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38798.64 MB 2025-02-15 07:25:47,944 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 07:25:47,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:47,945 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:25:47,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:47,946 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:25:47,951 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:25:47,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:25:47,952 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:25:47,952 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:27:06,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:06,958 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:27:06,963 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:27:06,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:06,968 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:27:06,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:06,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:27:23,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:27:23,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:27:23,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.00 seconds 2025-02-15 07:27:23,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:23,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33575.76 MB 2025-02-15 07:27:23,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37482.75 MB 2025-02-15 07:27:23,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-15 07:27:23,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52474.94 MB 2025-02-15 07:27:23,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43652.22 MB 2025-02-15 07:27:23,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8822.72 MB 2025-02-15 07:27:23,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46444.51 MB 2025-02-15 07:27:24,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:27:24,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:27:24,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 07:27:24,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:24,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37482.75 MB 2025-02-15 07:27:24,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33244.51 MB 2025-02-15 07:27:24,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4238.24 MB 2025-02-15 07:27:24,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43652.22 MB 2025-02-15 07:27:24,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43652.22 MB 2025-02-15 07:27:24,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:27:24,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40727.39 MB 2025-02-15 07:27:25,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:27:25,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:27:25,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.10 seconds 2025-02-15 07:27:25,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33244.51 MB 2025-02-15 07:27:25,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33551.07 MB 2025-02-15 07:27:25,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.56 MB 2025-02-15 07:27:25,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43652.22 MB 2025-02-15 07:27:25,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39745.22 MB 2025-02-15 07:27:25,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3906.99 MB 2025-02-15 07:27:25,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37499.10 MB 2025-02-15 07:27:25,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:27:25,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:27:25,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:27:25,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33551.07 MB 2025-02-15 07:27:25,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34642.02 MB 2025-02-15 07:27:25,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1090.94 MB 2025-02-15 07:27:25,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39745.22 MB 2025-02-15 07:27:25,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39745.22 MB 2025-02-15 07:27:25,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:27:25,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35460.58 MB 2025-02-15 07:27:25,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:27:25,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:27:25,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 07:27:25,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34642.02 MB 2025-02-15 07:27:25,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35936.71 MB 2025-02-15 07:27:25,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.70 MB 2025-02-15 07:27:25,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39745.22 MB 2025-02-15 07:27:25,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41926.26 MB 2025-02-15 07:27:25,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2181.04 MB 2025-02-15 07:27:25,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39139.56 MB 2025-02-15 07:27:25,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:27:25,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:27:25,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 07:27:25,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33551.07 MB 2025-02-15 07:27:25,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35936.71 MB 2025-02-15 07:27:25,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.64 MB 2025-02-15 07:27:25,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39745.22 MB 2025-02-15 07:27:25,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41926.26 MB 2025-02-15 07:27:25,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2181.04 MB 2025-02-15 07:27:25,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39139.56 MB 2025-02-15 07:27:25,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:27:25,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:27:25,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:27:25,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36822.33 MB 2025-02-15 07:27:25,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37265.28 MB 2025-02-15 07:27:25,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 442.94 MB 2025-02-15 07:27:25,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41926.26 MB 2025-02-15 07:27:25,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42161.14 MB 2025-02-15 07:27:25,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 234.88 MB 2025-02-15 07:27:25,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37674.03 MB 2025-02-15 07:27:25,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:27:25,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:27:25,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:27:25,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37503.73 MB 2025-02-15 07:27:25,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37725.30 MB 2025-02-15 07:27:25,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.57 MB 2025-02-15 07:27:25,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42161.14 MB 2025-02-15 07:27:25,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42161.14 MB 2025-02-15 07:27:25,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:27:25,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37797.46 MB 2025-02-15 07:27:25,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:27:25,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:27:25,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.43 seconds 2025-02-15 07:27:25,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.33 MB 2025-02-15 07:27:25,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37926.37 MB 2025-02-15 07:27:25,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8197.04 MB 2025-02-15 07:27:25,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52474.94 MB 2025-02-15 07:27:25,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42161.14 MB 2025-02-15 07:27:25,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10313.79 MB 2025-02-15 07:27:25,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37926.37 MB 2025-02-15 07:27:25,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:27:25,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:27:25,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:27:25,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30921.59 MB 2025-02-15 07:27:25,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33935.62 MB 2025-02-15 07:27:25,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 07:27:25,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42161.14 MB 2025-02-15 07:27:25,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42161.14 MB 2025-02-15 07:27:25,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:27:25,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34236.99 MB 2025-02-15 07:27:25,683 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:27:25,683 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:27:25,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:27:25,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:27:25,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:27:25,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:27:25,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33935.62 MB 2025-02-15 07:27:25,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42374.64 MB 2025-02-15 07:27:25,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:27:25,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42161.14 MB 2025-02-15 07:27:25,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50551.85 MB 2025-02-15 07:27:25,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:27:25,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42374.64 MB 2025-02-15 07:27:25,881 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:27:25,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:25,884 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:27:25,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:25,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:27:25,893 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:27:25,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:27:25,895 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:27:25,895 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:28:37,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:28:37,151 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:28:37,156 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:28:37,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:28:37,160 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1987, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:28:37,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:28:37,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1987, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:29:08,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:29:08,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:29:08,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.00 seconds 2025-02-15 07:29:08,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:08,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39728.64 MB 2025-02-15 07:29:08,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46760.52 MB 2025-02-15 07:29:08,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7031.88 MB 2025-02-15 07:29:08,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63136.86 MB 2025-02-15 07:29:08,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55165.58 MB 2025-02-15 07:29:08,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7971.27 MB 2025-02-15 07:29:08,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55768.29 MB 2025-02-15 07:29:08,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:29:08,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:29:08,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:29:08,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:08,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46760.52 MB 2025-02-15 07:29:08,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39021.85 MB 2025-02-15 07:29:08,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7738.68 MB 2025-02-15 07:29:08,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55165.58 MB 2025-02-15 07:29:08,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69449.29 MB 2025-02-15 07:29:08,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14283.70 MB 2025-02-15 07:29:08,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66116.14 MB 2025-02-15 07:29:10,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:29:10,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:29:10,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:29:10,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39021.85 MB 2025-02-15 07:29:10,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39552.69 MB 2025-02-15 07:29:10,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:29:10,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69449.29 MB 2025-02-15 07:29:10,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49547.31 MB 2025-02-15 07:29:10,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19901.97 MB 2025-02-15 07:29:10,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43531.23 MB 2025-02-15 07:29:10,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:29:10,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:29:10,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:29:10,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39552.69 MB 2025-02-15 07:29:10,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41442.22 MB 2025-02-15 07:29:10,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:29:10,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49547.31 MB 2025-02-15 07:29:10,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49547.31 MB 2025-02-15 07:29:10,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:29:10,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42859.65 MB 2025-02-15 07:29:10,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:29:10,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:29:10,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:29:10,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41442.22 MB 2025-02-15 07:29:10,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43684.08 MB 2025-02-15 07:29:10,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:29:10,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49547.31 MB 2025-02-15 07:29:10,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54265.91 MB 2025-02-15 07:29:10,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 07:29:10,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49228.36 MB 2025-02-15 07:29:10,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:29:10,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:29:10,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:29:10,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39552.69 MB 2025-02-15 07:29:10,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43684.08 MB 2025-02-15 07:29:10,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:29:10,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49547.31 MB 2025-02-15 07:29:10,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54265.91 MB 2025-02-15 07:29:10,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 07:29:10,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49228.36 MB 2025-02-15 07:29:10,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:29:10,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:29:10,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 07:29:10,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45217.62 MB 2025-02-15 07:29:10,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45984.62 MB 2025-02-15 07:29:10,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:29:10,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54265.91 MB 2025-02-15 07:29:10,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54679.04 MB 2025-02-15 07:29:10,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:29:10,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46692.41 MB 2025-02-15 07:29:10,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:29:10,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:29:10,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:29:10,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46397.51 MB 2025-02-15 07:29:10,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46626.03 MB 2025-02-15 07:29:10,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-15 07:29:10,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54679.04 MB 2025-02-15 07:29:10,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54679.04 MB 2025-02-15 07:29:10,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:29:10,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46825.95 MB 2025-02-15 07:29:10,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:29:10,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:29:10,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.61 seconds 2025-02-15 07:29:10,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:10,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32805.78 MB 2025-02-15 07:29:10,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46827.00 MB 2025-02-15 07:29:10,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14021.23 MB 2025-02-15 07:29:10,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63136.86 MB 2025-02-15 07:29:10,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54679.04 MB 2025-02-15 07:29:10,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8457.81 MB 2025-02-15 07:29:10,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46827.00 MB 2025-02-15 07:29:11,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:29:11,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:29:11,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:29:11,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:11,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46827.00 MB 2025-02-15 07:29:11,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37808.10 MB 2025-02-15 07:29:11,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9018.90 MB 2025-02-15 07:29:11,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54679.04 MB 2025-02-15 07:29:11,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54679.04 MB 2025-02-15 07:29:11,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:29:11,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49337.44 MB 2025-02-15 07:29:11,056 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 07:29:11,057 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:29:11,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:29:11,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:29:11,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:29:11,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:11,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37808.10 MB 2025-02-15 07:29:11,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46242.95 MB 2025-02-15 07:29:11,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 07:29:11,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54679.04 MB 2025-02-15 07:29:11,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63065.55 MB 2025-02-15 07:29:11,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 07:29:11,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46242.95 MB 2025-02-15 07:29:11,224 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 07:29:11,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:11,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:29:11,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:11,226 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:29:11,231 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:29:11,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:11,232 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:29:11,232 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:29:25,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:25,576 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:29:25,581 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:29:25,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:25,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:29:25,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:25,586 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:29:51,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:29:51,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:29:51,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.32 seconds 2025-02-15 07:29:51,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:51,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37568.51 MB 2025-02-15 07:29:51,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43503.45 MB 2025-02-15 07:29:51,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-15 07:29:51,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75644.27 MB 2025-02-15 07:29:51,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54064.58 MB 2025-02-15 07:29:51,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21579.69 MB 2025-02-15 07:29:51,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52475.70 MB 2025-02-15 07:29:52,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:29:52,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:29:52,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:29:52,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:52,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43503.45 MB 2025-02-15 07:29:52,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37410.25 MB 2025-02-15 07:29:52,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-15 07:29:52,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54064.58 MB 2025-02-15 07:29:52,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66817.36 MB 2025-02-15 07:29:52,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12752.78 MB 2025-02-15 07:29:52,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60554.17 MB 2025-02-15 07:29:53,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:29:53,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:29:53,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:29:53,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:53,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37410.25 MB 2025-02-15 07:29:53,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37941.09 MB 2025-02-15 07:29:53,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:29:53,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66817.36 MB 2025-02-15 07:29:53,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45353.01 MB 2025-02-15 07:29:53,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21464.35 MB 2025-02-15 07:29:53,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41919.64 MB 2025-02-15 07:29:53,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:29:53,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:29:53,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:29:53,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:53,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37941.09 MB 2025-02-15 07:29:53,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39830.63 MB 2025-02-15 07:29:53,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:29:53,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45353.01 MB 2025-02-15 07:29:53,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46296.73 MB 2025-02-15 07:29:53,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:29:53,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41248.06 MB 2025-02-15 07:29:54,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:29:54,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:29:54,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:29:54,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39830.63 MB 2025-02-15 07:29:54,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42072.48 MB 2025-02-15 07:29:54,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:29:54,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46296.73 MB 2025-02-15 07:29:54,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51959.04 MB 2025-02-15 07:29:54,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:29:54,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47616.77 MB 2025-02-15 07:29:54,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:29:54,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:29:54,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:29:54,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37941.09 MB 2025-02-15 07:29:54,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42072.48 MB 2025-02-15 07:29:54,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:29:54,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45353.01 MB 2025-02-15 07:29:54,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51959.04 MB 2025-02-15 07:29:54,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:29:54,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47616.77 MB 2025-02-15 07:29:54,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:29:54,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:29:54,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:29:54,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43606.03 MB 2025-02-15 07:29:54,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44373.03 MB 2025-02-15 07:29:54,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:29:54,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51959.04 MB 2025-02-15 07:29:54,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52370.08 MB 2025-02-15 07:29:54,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:29:54,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45080.82 MB 2025-02-15 07:29:54,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:29:54,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:29:54,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:29:54,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44785.92 MB 2025-02-15 07:29:54,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45014.16 MB 2025-02-15 07:29:54,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-15 07:29:54,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52370.08 MB 2025-02-15 07:29:54,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52370.08 MB 2025-02-15 07:29:54,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:29:54,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45254.90 MB 2025-02-15 07:29:54,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:29:54,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:29:54,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.79 seconds 2025-02-15 07:29:54,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31725.71 MB 2025-02-15 07:29:54,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45215.02 MB 2025-02-15 07:29:54,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13489.31 MB 2025-02-15 07:29:54,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75644.27 MB 2025-02-15 07:29:54,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52370.08 MB 2025-02-15 07:29:54,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23274.19 MB 2025-02-15 07:29:54,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45254.90 MB 2025-02-15 07:29:54,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:29:54,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:29:54,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:29:54,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45215.02 MB 2025-02-15 07:29:54,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36723.99 MB 2025-02-15 07:29:54,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8491.02 MB 2025-02-15 07:29:54,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52370.08 MB 2025-02-15 07:29:54,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52370.08 MB 2025-02-15 07:29:54,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:29:54,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47722.08 MB 2025-02-15 07:29:54,663 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 07:29:54,663 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:29:54,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:29:54,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:29:54,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:29:54,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:29:54,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36723.99 MB 2025-02-15 07:29:54,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45147.20 MB 2025-02-15 07:29:54,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 07:29:54,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52370.08 MB 2025-02-15 07:29:54,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60746.10 MB 2025-02-15 07:29:54,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 07:29:54,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45147.20 MB 2025-02-15 07:29:54,837 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 07:29:54,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:54,838 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:29:54,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:54,839 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:29:54,844 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:29:54,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:29:54,845 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:29:54,845 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:30:49,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:49,963 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:30:49,968 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:30:49,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:49,971 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:30:49,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:49,972 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:30:55,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:30:55,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:30:55,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.15 seconds 2025-02-15 07:30:55,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:55,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28196.34 MB 2025-02-15 07:30:55,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29371.27 MB 2025-02-15 07:30:55,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1174.93 MB 2025-02-15 07:30:55,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69122.13 MB 2025-02-15 07:30:55,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36500.93 MB 2025-02-15 07:30:55,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32621.20 MB 2025-02-15 07:30:55,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38347.19 MB 2025-02-15 07:30:55,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:30:55,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:30:55,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:30:55,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:55,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29371.27 MB 2025-02-15 07:30:55,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29940.45 MB 2025-02-15 07:30:55,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.18 MB 2025-02-15 07:30:55,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36500.93 MB 2025-02-15 07:30:55,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37677.43 MB 2025-02-15 07:30:55,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1176.50 MB 2025-02-15 07:30:55,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34034.55 MB 2025-02-15 07:30:56,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:30:56,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:30:56,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-15 07:30:56,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:56,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29940.45 MB 2025-02-15 07:30:56,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30381.05 MB 2025-02-15 07:30:56,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-15 07:30:56,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37677.43 MB 2025-02-15 07:30:56,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37677.43 MB 2025-02-15 07:30:56,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:30:56,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34364.91 MB 2025-02-15 07:30:56,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:30:56,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:30:56,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:30:56,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:56,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30381.05 MB 2025-02-15 07:30:56,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31949.18 MB 2025-02-15 07:30:56,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1568.13 MB 2025-02-15 07:30:56,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37677.43 MB 2025-02-15 07:30:56,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37677.43 MB 2025-02-15 07:30:56,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:30:56,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33125.65 MB 2025-02-15 07:30:56,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:30:56,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:30:56,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:30:56,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:56,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31949.18 MB 2025-02-15 07:30:56,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33809.93 MB 2025-02-15 07:30:56,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1860.75 MB 2025-02-15 07:30:56,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37677.43 MB 2025-02-15 07:30:56,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41599.11 MB 2025-02-15 07:30:56,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3921.67 MB 2025-02-15 07:30:56,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38415.87 MB 2025-02-15 07:30:56,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:30:56,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:30:56,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 07:30:56,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:56,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30381.05 MB 2025-02-15 07:30:56,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33809.93 MB 2025-02-15 07:30:56,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3428.88 MB 2025-02-15 07:30:56,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37677.43 MB 2025-02-15 07:30:56,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41599.11 MB 2025-02-15 07:30:56,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3921.67 MB 2025-02-15 07:30:56,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38415.87 MB 2025-02-15 07:30:57,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:30:57,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:30:57,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 07:30:57,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:57,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35082.77 MB 2025-02-15 07:30:57,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35719.38 MB 2025-02-15 07:30:57,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 636.61 MB 2025-02-15 07:30:57,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41599.11 MB 2025-02-15 07:30:57,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41938.85 MB 2025-02-15 07:30:57,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 339.74 MB 2025-02-15 07:30:57,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36306.85 MB 2025-02-15 07:30:57,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:30:57,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:30:57,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:30:57,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:57,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36062.08 MB 2025-02-15 07:30:57,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36276.32 MB 2025-02-15 07:30:57,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.23 MB 2025-02-15 07:30:57,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41938.85 MB 2025-02-15 07:30:57,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41938.85 MB 2025-02-15 07:30:57,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:30:57,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36424.97 MB 2025-02-15 07:30:57,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:30:57,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:30:57,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.11 seconds 2025-02-15 07:30:57,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:57,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27039.62 MB 2025-02-15 07:30:57,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36477.39 MB 2025-02-15 07:30:57,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9437.77 MB 2025-02-15 07:30:57,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69122.13 MB 2025-02-15 07:30:57,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41938.85 MB 2025-02-15 07:30:57,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27183.28 MB 2025-02-15 07:30:57,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36477.39 MB 2025-02-15 07:30:57,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:30:57,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:30:57,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:30:57,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:57,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36477.39 MB 2025-02-15 07:30:57,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39491.42 MB 2025-02-15 07:30:57,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 07:30:57,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41938.85 MB 2025-02-15 07:30:57,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41938.85 MB 2025-02-15 07:30:57,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:30:57,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39792.79 MB 2025-02-15 07:30:57,371 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:30:57,371 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:30:57,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:30:57,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:30:57,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:30:57,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:30:57,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31722.58 MB 2025-02-15 07:30:57,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40161.60 MB 2025-02-15 07:30:57,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:30:57,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41938.85 MB 2025-02-15 07:30:57,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52428.80 MB 2025-02-15 07:30:57,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 07:30:57,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40161.60 MB 2025-02-15 07:30:57,539 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:30:57,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:57,540 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:30:57,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:57,541 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:30:57,546 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:30:57,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:30:57,547 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:30:57,547 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:32:21,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:21,874 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:32:21,882 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:32:21,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:21,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:32:21,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:21,891 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:32:41,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:32:41,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:32:41,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.94 seconds 2025-02-15 07:32:41,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:41,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21957.63 MB 2025-02-15 07:32:41,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26523.07 MB 2025-02-15 07:32:41,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4565.43 MB 2025-02-15 07:32:41,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65013.81 MB 2025-02-15 07:32:41,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37776.00 MB 2025-02-15 07:32:41,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27237.81 MB 2025-02-15 07:32:41,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.80 MB 2025-02-15 07:32:41,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:32:41,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:32:41,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:32:41,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:41,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.07 MB 2025-02-15 07:32:41,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.16 MB 2025-02-15 07:32:41,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4038.91 MB 2025-02-15 07:32:41,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37776.00 MB 2025-02-15 07:32:41,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43583.01 MB 2025-02-15 07:32:41,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5807.01 MB 2025-02-15 07:32:41,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38509.02 MB 2025-02-15 07:32:43,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:32:43,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:32:43,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:32:43,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:43,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.16 MB 2025-02-15 07:32:43,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23015.00 MB 2025-02-15 07:32:43,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:32:43,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43583.01 MB 2025-02-15 07:32:43,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33210.50 MB 2025-02-15 07:32:43,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10372.51 MB 2025-02-15 07:32:43,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.55 MB 2025-02-15 07:32:43,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:32:43,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:32:43,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:32:43,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:43,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-15 07:32:43,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.53 MB 2025-02-15 07:32:43,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:32:43,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33210.50 MB 2025-02-15 07:32:43,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33210.50 MB 2025-02-15 07:32:43,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:32:43,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26321.96 MB 2025-02-15 07:32:44,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:32:44,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:32:44,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:32:44,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.53 MB 2025-02-15 07:32:44,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-15 07:32:44,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:32:44,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33210.50 MB 2025-02-15 07:32:44,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34154.22 MB 2025-02-15 07:32:44,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:32:44,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-15 07:32:44,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:32:44,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:32:44,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:32:44,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-15 07:32:44,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-15 07:32:44,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:32:44,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33210.50 MB 2025-02-15 07:32:44,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34154.22 MB 2025-02-15 07:32:44,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:32:44,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-15 07:32:44,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:32:44,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:32:44,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 07:32:44,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28679.93 MB 2025-02-15 07:32:44,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29446.93 MB 2025-02-15 07:32:44,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:32:44,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34154.22 MB 2025-02-15 07:32:44,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-15 07:32:44,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:32:44,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.72 MB 2025-02-15 07:32:44,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:32:44,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:32:44,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:32:44,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29859.82 MB 2025-02-15 07:32:44,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30089.55 MB 2025-02-15 07:32:44,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.72 MB 2025-02-15 07:32:44,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34567.36 MB 2025-02-15 07:32:44,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-15 07:32:44,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:32:44,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30291.18 MB 2025-02-15 07:32:44,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:32:44,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:32:44,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.38 seconds 2025-02-15 07:32:44,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.17 MB 2025-02-15 07:32:44,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30290.50 MB 2025-02-15 07:32:44,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12827.33 MB 2025-02-15 07:32:44,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65013.81 MB 2025-02-15 07:32:44,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-15 07:32:44,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30446.45 MB 2025-02-15 07:32:44,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30291.18 MB 2025-02-15 07:32:44,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:32:44,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:32:44,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:32:44,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19453.46 MB 2025-02-15 07:32:44,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22465.65 MB 2025-02-15 07:32:44,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.19 MB 2025-02-15 07:32:44,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34567.36 MB 2025-02-15 07:32:44,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-15 07:32:44,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:32:44,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22766.84 MB 2025-02-15 07:32:44,561 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 07:32:44,561 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:32:44,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:32:44,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:32:44,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:32:44,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:32:44,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22465.65 MB 2025-02-15 07:32:44,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30900.27 MB 2025-02-15 07:32:44,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 07:32:44,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34567.36 MB 2025-02-15 07:32:44,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42951.77 MB 2025-02-15 07:32:44,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 07:32:44,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30900.27 MB 2025-02-15 07:32:44,730 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 07:32:44,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:44,731 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:32:44,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:44,732 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:32:44,737 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:32:44,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:32:44,738 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:32:44,738 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:33:17,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:17,723 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:33:17,728 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:33:17,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:17,732 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1780, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:33:17,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:17,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1780, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:33:45,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:33:45,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:33:45,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.79 seconds 2025-02-15 07:33:45,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:45,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25372.03 MB 2025-02-15 07:33:45,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31671.88 MB 2025-02-15 07:33:45,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6299.84 MB 2025-02-15 07:33:45,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51336.18 MB 2025-02-15 07:33:45,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39510.34 MB 2025-02-15 07:33:45,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11825.84 MB 2025-02-15 07:33:45,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40505.71 MB 2025-02-15 07:33:45,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:33:45,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:33:45,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 07:33:45,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:45,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31671.88 MB 2025-02-15 07:33:45,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25031.52 MB 2025-02-15 07:33:45,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6640.36 MB 2025-02-15 07:33:45,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39510.34 MB 2025-02-15 07:33:45,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53150.22 MB 2025-02-15 07:33:45,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13639.88 MB 2025-02-15 07:33:45,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50051.59 MB 2025-02-15 07:33:47,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:33:47,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:33:47,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:33:47,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:47,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25031.52 MB 2025-02-15 07:33:47,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25562.36 MB 2025-02-15 07:33:47,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:33:47,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53150.22 MB 2025-02-15 07:33:47,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34626.08 MB 2025-02-15 07:33:47,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18524.14 MB 2025-02-15 07:33:47,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29540.90 MB 2025-02-15 07:33:47,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:33:47,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:33:47,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:33:47,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:47,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-15 07:33:47,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27451.89 MB 2025-02-15 07:33:47,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:33:47,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34626.08 MB 2025-02-15 07:33:47,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34626.08 MB 2025-02-15 07:33:47,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:33:47,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.32 MB 2025-02-15 07:33:47,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:33:47,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:33:47,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:33:47,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:47,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27451.89 MB 2025-02-15 07:33:47,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-15 07:33:47,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:33:47,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34626.08 MB 2025-02-15 07:33:47,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37457.23 MB 2025-02-15 07:33:47,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:33:47,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-15 07:33:47,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:33:47,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:33:47,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:33:47,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:47,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-15 07:33:47,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-15 07:33:47,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:33:47,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34626.08 MB 2025-02-15 07:33:47,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37457.23 MB 2025-02-15 07:33:47,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:33:47,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-15 07:33:48,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:33:48,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:33:48,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:33:48,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:48,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31227.29 MB 2025-02-15 07:33:48,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31994.29 MB 2025-02-15 07:33:48,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:33:48,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37457.23 MB 2025-02-15 07:33:48,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-15 07:33:48,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 07:33:48,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32702.08 MB 2025-02-15 07:33:48,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:33:48,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:33:48,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:33:48,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:48,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.18 MB 2025-02-15 07:33:48,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32635.76 MB 2025-02-15 07:33:48,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.58 MB 2025-02-15 07:33:48,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-15 07:33:48,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-15 07:33:48,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:33:48,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32844.76 MB 2025-02-15 07:33:48,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:33:48,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:33:48,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.29 seconds 2025-02-15 07:33:48,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:48,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19170.37 MB 2025-02-15 07:33:48,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32836.05 MB 2025-02-15 07:33:48,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13665.68 MB 2025-02-15 07:33:48,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51336.18 MB 2025-02-15 07:33:48,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-15 07:33:48,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13467.91 MB 2025-02-15 07:33:48,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32844.76 MB 2025-02-15 07:33:48,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:33:48,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:33:48,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:33:48,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:48,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32836.05 MB 2025-02-15 07:33:48,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24163.13 MB 2025-02-15 07:33:48,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8672.91 MB 2025-02-15 07:33:48,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-15 07:33:48,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-15 07:33:48,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:33:48,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35338.45 MB 2025-02-15 07:33:48,314 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 07:33:48,315 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:33:48,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:33:48,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:33:48,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:33:48,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:33:48,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24163.13 MB 2025-02-15 07:33:48,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32568.79 MB 2025-02-15 07:33:48,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 07:33:48,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-15 07:33:48,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46227.52 MB 2025-02-15 07:33:48,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 07:33:48,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32568.79 MB 2025-02-15 07:33:48,478 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 07:33:48,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:48,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:33:48,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:48,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:33:48,485 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:33:48,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:33:48,486 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:33:48,486 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:34:22,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:22,015 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:34:22,022 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:34:22,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:22,028 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 734, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:34:22,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:22,030 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 734, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:34:33,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:34:33,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:34:33,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.57 seconds 2025-02-15 07:34:33,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:33,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18083.34 MB 2025-02-15 07:34:33,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20681.71 MB 2025-02-15 07:34:33,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2598.37 MB 2025-02-15 07:34:33,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54586.77 MB 2025-02-15 07:34:33,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24685.58 MB 2025-02-15 07:34:33,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29901.19 MB 2025-02-15 07:34:33,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29593.95 MB 2025-02-15 07:34:33,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:34:33,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:34:33,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:34:33,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:33,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20681.71 MB 2025-02-15 07:34:33,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19594.74 MB 2025-02-15 07:34:33,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1086.97 MB 2025-02-15 07:34:33,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24685.58 MB 2025-02-15 07:34:33,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31396.46 MB 2025-02-15 07:34:33,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6710.89 MB 2025-02-15 07:34:33,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29494.01 MB 2025-02-15 07:34:35,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:34:35,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:34:35,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:34:35,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:35,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19594.74 MB 2025-02-15 07:34:35,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20125.58 MB 2025-02-15 07:34:35,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:34:35,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31396.46 MB 2025-02-15 07:34:35,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24211.62 MB 2025-02-15 07:34:35,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7184.84 MB 2025-02-15 07:34:35,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24105.16 MB 2025-02-15 07:34:35,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:34:35,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:34:35,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:34:35,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:35,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-15 07:34:35,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22015.11 MB 2025-02-15 07:34:35,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:34:35,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24211.62 MB 2025-02-15 07:34:35,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25155.34 MB 2025-02-15 07:34:35,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:34:35,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23432.54 MB 2025-02-15 07:34:35,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:34:35,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:34:35,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:34:35,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:35,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22015.11 MB 2025-02-15 07:34:35,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-15 07:34:35,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:34:35,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25155.34 MB 2025-02-15 07:34:35,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31761.37 MB 2025-02-15 07:34:35,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:34:35,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-15 07:34:35,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:34:35,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:34:35,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:34:35,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:35,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-15 07:34:35,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-15 07:34:35,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:34:35,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24211.62 MB 2025-02-15 07:34:35,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31761.37 MB 2025-02-15 07:34:35,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 07:34:35,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-15 07:34:36,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:34:36,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:34:36,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:34:36,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:36,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25790.51 MB 2025-02-15 07:34:36,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26557.51 MB 2025-02-15 07:34:36,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:34:36,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31761.37 MB 2025-02-15 07:34:36,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32176.60 MB 2025-02-15 07:34:36,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:34:36,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27265.30 MB 2025-02-15 07:34:36,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:34:36,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:34:36,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:34:36,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:36,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26970.40 MB 2025-02-15 07:34:36,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27200.42 MB 2025-02-15 07:34:36,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.02 MB 2025-02-15 07:34:36,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32176.60 MB 2025-02-15 07:34:36,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32176.60 MB 2025-02-15 07:34:36,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:34:36,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27408.02 MB 2025-02-15 07:34:36,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:34:36,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:34:36,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.00 seconds 2025-02-15 07:34:36,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:36,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.02 MB 2025-02-15 07:34:36,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27401.49 MB 2025-02-15 07:34:36,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11875.47 MB 2025-02-15 07:34:36,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54586.77 MB 2025-02-15 07:34:36,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32176.60 MB 2025-02-15 07:34:36,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22410.17 MB 2025-02-15 07:34:36,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27408.02 MB 2025-02-15 07:34:36,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:34:36,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:34:36,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:34:36,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:36,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27401.49 MB 2025-02-15 07:34:36,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20530.41 MB 2025-02-15 07:34:36,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6871.08 MB 2025-02-15 07:34:36,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32176.60 MB 2025-02-15 07:34:36,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32176.60 MB 2025-02-15 07:34:36,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:34:36,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29913.16 MB 2025-02-15 07:34:36,325 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:34:36,326 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:34:36,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:34:36,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:34:36,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:34:36,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:34:36,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20530.41 MB 2025-02-15 07:34:36,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28969.43 MB 2025-02-15 07:34:36,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:34:36,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32176.60 MB 2025-02-15 07:34:36,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40567.31 MB 2025-02-15 07:34:36,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:34:36,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28969.43 MB 2025-02-15 07:34:36,491 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:34:36,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:36,492 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:34:36,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:36,493 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:34:36,498 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:34:36,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:34:36,499 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:34:36,499 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 07:36:19,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:19,501 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:36:19,507 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:36:19,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:19,511 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 675, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:36:19,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:19,512 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 675, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:36:29,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:36:29,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:36:29,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.44 seconds 2025-02-15 07:36:29,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:29,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.21 MB 2025-02-15 07:36:29,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20061.00 MB 2025-02-15 07:36:29,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2388.79 MB 2025-02-15 07:36:29,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53152.32 MB 2025-02-15 07:36:29,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23502.78 MB 2025-02-15 07:36:29,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29649.53 MB 2025-02-15 07:36:29,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28955.52 MB 2025-02-15 07:36:30,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:36:30,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:36:30,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:36:30,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:30,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20061.00 MB 2025-02-15 07:36:30,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19288.01 MB 2025-02-15 07:36:30,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -772.99 MB 2025-02-15 07:36:30,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23502.78 MB 2025-02-15 07:36:30,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29590.81 MB 2025-02-15 07:36:30,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6088.03 MB 2025-02-15 07:36:30,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28248.61 MB 2025-02-15 07:36:31,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:36:31,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:36:31,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:36:31,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:31,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.01 MB 2025-02-15 07:36:31,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19818.86 MB 2025-02-15 07:36:31,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:36:31,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29590.81 MB 2025-02-15 07:36:31,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24211.62 MB 2025-02-15 07:36:31,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5379.19 MB 2025-02-15 07:36:31,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23797.40 MB 2025-02-15 07:36:31,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:36:31,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:36:31,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:36:31,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:31,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19818.86 MB 2025-02-15 07:36:31,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21708.39 MB 2025-02-15 07:36:31,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:36:31,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24211.62 MB 2025-02-15 07:36:31,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25155.34 MB 2025-02-15 07:36:31,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:36:31,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23125.82 MB 2025-02-15 07:36:32,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:36:32,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:36:32,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:36:32,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21708.39 MB 2025-02-15 07:36:32,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.25 MB 2025-02-15 07:36:32,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:36:32,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25155.34 MB 2025-02-15 07:36:32,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31289.51 MB 2025-02-15 07:36:32,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 07:36:32,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29494.53 MB 2025-02-15 07:36:32,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:36:32,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:36:32,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:36:32,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19818.86 MB 2025-02-15 07:36:32,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.25 MB 2025-02-15 07:36:32,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:36:32,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24211.62 MB 2025-02-15 07:36:32,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31289.51 MB 2025-02-15 07:36:32,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 07:36:32,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29494.53 MB 2025-02-15 07:36:32,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:36:32,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:36:32,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:36:32,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25483.79 MB 2025-02-15 07:36:32,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26250.79 MB 2025-02-15 07:36:32,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:36:32,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31289.51 MB 2025-02-15 07:36:32,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31704.74 MB 2025-02-15 07:36:32,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:36:32,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26958.58 MB 2025-02-15 07:36:32,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:36:32,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:36:32,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:36:32,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26663.68 MB 2025-02-15 07:36:32,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26893.38 MB 2025-02-15 07:36:32,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.70 MB 2025-02-15 07:36:32,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31704.74 MB 2025-02-15 07:36:32,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31704.74 MB 2025-02-15 07:36:32,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:36:32,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27090.24 MB 2025-02-15 07:36:32,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:36:32,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:36:32,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.84 seconds 2025-02-15 07:36:32,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15320.46 MB 2025-02-15 07:36:32,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27094.45 MB 2025-02-15 07:36:32,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11773.99 MB 2025-02-15 07:36:32,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53152.32 MB 2025-02-15 07:36:32,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31704.74 MB 2025-02-15 07:36:32,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21447.57 MB 2025-02-15 07:36:32,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27094.45 MB 2025-02-15 07:36:32,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:36:32,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:36:32,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:36:32,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27094.45 MB 2025-02-15 07:36:32,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20324.85 MB 2025-02-15 07:36:32,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6769.60 MB 2025-02-15 07:36:32,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31704.74 MB 2025-02-15 07:36:32,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31704.74 MB 2025-02-15 07:36:32,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:36:32,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29606.12 MB 2025-02-15 07:36:32,643 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:36:32,644 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:36:32,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:36:32,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:36:32,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:36:32,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:36:32,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20324.85 MB 2025-02-15 07:36:32,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28763.87 MB 2025-02-15 07:36:32,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:36:32,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31704.74 MB 2025-02-15 07:36:32,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40095.45 MB 2025-02-15 07:36:32,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:36:32,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28763.87 MB 2025-02-15 07:36:32,813 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:36:32,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:32,815 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:36:32,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:32,815 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:36:32,820 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:36:32,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:36:32,821 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:36:32,821 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:37:53,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:37:53,886 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:37:53,891 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:37:53,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:37:53,895 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2564, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:37:53,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:37:53,896 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2564, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:38:33,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:38:33,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:38:33,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.74 seconds 2025-02-15 07:38:33,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:33,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30835.07 MB 2025-02-15 07:38:33,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39909.45 MB 2025-02-15 07:38:33,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9074.38 MB 2025-02-15 07:38:33,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70552.39 MB 2025-02-15 07:38:33,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43425.73 MB 2025-02-15 07:38:33,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27126.66 MB 2025-02-15 07:38:33,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48983.30 MB 2025-02-15 07:38:33,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:38:33,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:38:33,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 07:38:33,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:33,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39909.45 MB 2025-02-15 07:38:33,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29108.34 MB 2025-02-15 07:38:33,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10801.11 MB 2025-02-15 07:38:33,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43425.73 MB 2025-02-15 07:38:33,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63237.52 MB 2025-02-15 07:38:33,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19811.79 MB 2025-02-15 07:38:33,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66509.45 MB 2025-02-15 07:38:35,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:38:35,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:38:35,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:38:35,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:35,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29108.34 MB 2025-02-15 07:38:35,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.18 MB 2025-02-15 07:38:35,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:38:35,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63237.52 MB 2025-02-15 07:38:35,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31937.53 MB 2025-02-15 07:38:35,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31299.99 MB 2025-02-15 07:38:35,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.72 MB 2025-02-15 07:38:35,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:38:35,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:38:35,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:38:35,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:35,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29639.18 MB 2025-02-15 07:38:35,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31528.71 MB 2025-02-15 07:38:35,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:38:35,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31937.53 MB 2025-02-15 07:38:35,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34768.68 MB 2025-02-15 07:38:35,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:38:35,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32946.14 MB 2025-02-15 07:38:36,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:38:36,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:38:36,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:38:36,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31528.71 MB 2025-02-15 07:38:36,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33770.57 MB 2025-02-15 07:38:36,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:38:36,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34768.68 MB 2025-02-15 07:38:36,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40902.85 MB 2025-02-15 07:38:36,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 07:38:36,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39314.85 MB 2025-02-15 07:38:36,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:38:36,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:38:36,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 07:38:36,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29639.18 MB 2025-02-15 07:38:36,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33770.57 MB 2025-02-15 07:38:36,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:38:36,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31937.53 MB 2025-02-15 07:38:36,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40902.85 MB 2025-02-15 07:38:36,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 07:38:36,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39314.85 MB 2025-02-15 07:38:36,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:38:36,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:38:36,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:38:36,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35304.11 MB 2025-02-15 07:38:36,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36071.11 MB 2025-02-15 07:38:36,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:38:36,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40902.85 MB 2025-02-15 07:38:36,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 07:38:36,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:38:36,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36778.90 MB 2025-02-15 07:38:36,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:38:36,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:38:36,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:38:36,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36484.00 MB 2025-02-15 07:38:36,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36711.80 MB 2025-02-15 07:38:36,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.80 MB 2025-02-15 07:38:36,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 07:38:36,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 07:38:36,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:38:36,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36915.74 MB 2025-02-15 07:38:36,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:38:36,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:38:36,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.37 seconds 2025-02-15 07:38:36,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21901.89 MB 2025-02-15 07:38:36,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36912.13 MB 2025-02-15 07:38:36,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15010.24 MB 2025-02-15 07:38:36,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61616.42 MB 2025-02-15 07:38:36,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 07:38:36,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20298.33 MB 2025-02-15 07:38:36,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36915.74 MB 2025-02-15 07:38:36,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:38:36,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:38:36,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:38:36,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36912.13 MB 2025-02-15 07:38:36,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26894.74 MB 2025-02-15 07:38:36,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10017.39 MB 2025-02-15 07:38:36,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 07:38:36,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 07:38:36,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:38:36,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39414.58 MB 2025-02-15 07:38:36,558 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 07:38:36,558 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:38:36,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:38:36,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:38:36,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:38:36,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:38:36,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26894.74 MB 2025-02-15 07:38:36,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35303.00 MB 2025-02-15 07:38:36,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.25 MB 2025-02-15 07:38:36,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 07:38:36,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45497.71 MB 2025-02-15 07:38:36,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 07:38:36,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35303.00 MB 2025-02-15 07:38:36,724 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 07:38:36,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:38:36,726 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:38:36,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:38:36,727 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:38:36,731 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:38:36,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:38:36,732 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:38:36,732 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:39:44,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:39:44,905 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:39:44,913 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:39:44,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:39:44,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1792, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:39:44,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:39:44,923 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1792, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:40:12,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:40:12,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:40:12,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.01 seconds 2025-02-15 07:40:12,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:12,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25455.65 MB 2025-02-15 07:40:12,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31797.44 MB 2025-02-15 07:40:12,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6341.79 MB 2025-02-15 07:40:12,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53856.96 MB 2025-02-15 07:40:12,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39759.90 MB 2025-02-15 07:40:12,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14097.06 MB 2025-02-15 07:40:12,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40815.82 MB 2025-02-15 07:40:13,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:40:13,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:40:13,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 07:40:13,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:13,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31797.44 MB 2025-02-15 07:40:13,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25093.90 MB 2025-02-15 07:40:13,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6703.54 MB 2025-02-15 07:40:13,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39759.90 MB 2025-02-15 07:40:13,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53536.10 MB 2025-02-15 07:40:13,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13776.19 MB 2025-02-15 07:40:13,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50386.65 MB 2025-02-15 07:40:15,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:40:15,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:40:15,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:40:15,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25093.90 MB 2025-02-15 07:40:15,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25624.74 MB 2025-02-15 07:40:15,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:40:15,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53536.10 MB 2025-02-15 07:40:15,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30654.07 MB 2025-02-15 07:40:15,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22882.03 MB 2025-02-15 07:40:15,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29603.29 MB 2025-02-15 07:40:15,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:40:15,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:40:15,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:40:15,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.74 MB 2025-02-15 07:40:15,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27514.27 MB 2025-02-15 07:40:15,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:40:15,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-15 07:40:15,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30654.07 MB 2025-02-15 07:40:15,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:40:15,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28931.70 MB 2025-02-15 07:40:15,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:40:15,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:40:15,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:40:15,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27514.27 MB 2025-02-15 07:40:15,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29756.13 MB 2025-02-15 07:40:15,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:40:15,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-15 07:40:15,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37260.10 MB 2025-02-15 07:40:15,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:40:15,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35300.41 MB 2025-02-15 07:40:15,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:40:15,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:40:15,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:40:15,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.74 MB 2025-02-15 07:40:15,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29756.13 MB 2025-02-15 07:40:15,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:40:15,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-15 07:40:15,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37260.10 MB 2025-02-15 07:40:15,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:40:15,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35300.41 MB 2025-02-15 07:40:15,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:40:15,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:40:15,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:40:15,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31289.67 MB 2025-02-15 07:40:15,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32056.67 MB 2025-02-15 07:40:15,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:40:15,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37260.10 MB 2025-02-15 07:40:15,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37673.24 MB 2025-02-15 07:40:15,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:40:15,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32764.46 MB 2025-02-15 07:40:15,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:40:15,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:40:15,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:40:15,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32469.56 MB 2025-02-15 07:40:15,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32698.91 MB 2025-02-15 07:40:15,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.34 MB 2025-02-15 07:40:15,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37673.24 MB 2025-02-15 07:40:15,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37673.24 MB 2025-02-15 07:40:15,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:40:15,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.42 MB 2025-02-15 07:40:15,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:40:15,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:40:15,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.54 seconds 2025-02-15 07:40:15,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19212.18 MB 2025-02-15 07:40:15,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32899.76 MB 2025-02-15 07:40:15,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13687.58 MB 2025-02-15 07:40:15,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53856.96 MB 2025-02-15 07:40:15,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37673.24 MB 2025-02-15 07:40:15,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16183.72 MB 2025-02-15 07:40:15,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.42 MB 2025-02-15 07:40:15,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:40:15,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:40:15,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:40:15,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32899.76 MB 2025-02-15 07:40:15,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24199.95 MB 2025-02-15 07:40:15,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8699.80 MB 2025-02-15 07:40:15,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37673.24 MB 2025-02-15 07:40:15,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37673.24 MB 2025-02-15 07:40:15,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:40:15,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35397.29 MB 2025-02-15 07:40:15,766 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 07:40:15,767 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 07:40:15,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:40:15,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:40:15,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 07:40:15,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:40:15,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24199.95 MB 2025-02-15 07:40:15,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32591.51 MB 2025-02-15 07:40:15,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.56 MB 2025-02-15 07:40:15,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37673.24 MB 2025-02-15 07:40:15,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41844.47 MB 2025-02-15 07:40:15,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 07:40:15,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32591.51 MB 2025-02-15 07:40:15,941 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 07:40:15,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:40:15,942 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:40:15,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:40:15,943 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:40:15,948 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:40:15,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:40:15,949 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:40:15,949 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 07:41:04,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:04,873 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:41:04,879 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:41:04,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:04,883 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1434, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:41:04,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:04,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1434, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:41:27,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:41:27,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:41:27,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.33 seconds 2025-02-15 07:41:27,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:27,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22961.05 MB 2025-02-15 07:41:27,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28036.16 MB 2025-02-15 07:41:27,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5075.11 MB 2025-02-15 07:41:27,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50186.94 MB 2025-02-15 07:41:27,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38247.86 MB 2025-02-15 07:41:27,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11939.09 MB 2025-02-15 07:41:27,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36962.27 MB 2025-02-15 07:41:27,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:41:27,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:41:27,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:41:27,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:27,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28036.16 MB 2025-02-15 07:41:27,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23232.77 MB 2025-02-15 07:41:27,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4803.39 MB 2025-02-15 07:41:27,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38247.86 MB 2025-02-15 07:41:27,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48209.33 MB 2025-02-15 07:41:27,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9961.47 MB 2025-02-15 07:41:27,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43013.33 MB 2025-02-15 07:41:29,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:41:29,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:41:29,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 07:41:29,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23232.77 MB 2025-02-15 07:41:29,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23763.61 MB 2025-02-15 07:41:29,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:41:29,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48209.33 MB 2025-02-15 07:41:29,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29001.52 MB 2025-02-15 07:41:29,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19207.82 MB 2025-02-15 07:41:29,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27743.11 MB 2025-02-15 07:41:29,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:41:29,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:41:29,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:41:29,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-15 07:41:29,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25653.14 MB 2025-02-15 07:41:29,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:41:29,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29001.52 MB 2025-02-15 07:41:29,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29945.23 MB 2025-02-15 07:41:29,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:41:29,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27070.57 MB 2025-02-15 07:41:29,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:41:29,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:41:29,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:41:29,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25653.14 MB 2025-02-15 07:41:29,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-15 07:41:29,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:41:29,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29945.23 MB 2025-02-15 07:41:29,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35607.54 MB 2025-02-15 07:41:29,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:41:29,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-15 07:41:29,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:41:29,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:41:29,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:41:29,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-15 07:41:29,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-15 07:41:29,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:41:29,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29001.52 MB 2025-02-15 07:41:29,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35607.54 MB 2025-02-15 07:41:29,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:41:29,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-15 07:41:29,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:41:29,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:41:29,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:41:29,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29428.54 MB 2025-02-15 07:41:29,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30195.54 MB 2025-02-15 07:41:29,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:41:29,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35607.54 MB 2025-02-15 07:41:29,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36022.78 MB 2025-02-15 07:41:29,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:41:29,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30903.33 MB 2025-02-15 07:41:29,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:41:29,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:41:29,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:41:29,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30608.43 MB 2025-02-15 07:41:29,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30837.37 MB 2025-02-15 07:41:29,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-15 07:41:29,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36022.78 MB 2025-02-15 07:41:29,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36022.78 MB 2025-02-15 07:41:29,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:41:29,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31079.32 MB 2025-02-15 07:41:29,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:41:29,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:41:29,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.82 seconds 2025-02-15 07:41:29,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17964.88 MB 2025-02-15 07:41:29,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31038.22 MB 2025-02-15 07:41:29,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13073.34 MB 2025-02-15 07:41:29,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50186.94 MB 2025-02-15 07:41:29,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36022.78 MB 2025-02-15 07:41:29,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14164.16 MB 2025-02-15 07:41:29,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31079.32 MB 2025-02-15 07:41:29,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:41:29,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:41:29,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:41:29,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:29,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31038.22 MB 2025-02-15 07:41:29,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22952.30 MB 2025-02-15 07:41:29,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8085.92 MB 2025-02-15 07:41:29,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36022.78 MB 2025-02-15 07:41:29,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36022.78 MB 2025-02-15 07:41:29,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:41:29,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33535.45 MB 2025-02-15 07:41:29,998 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 07:41:29,998 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:41:30,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:41:30,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:41:30,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:41:30,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:41:30,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22952.30 MB 2025-02-15 07:41:30,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31343.34 MB 2025-02-15 07:41:30,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 07:41:30,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36022.78 MB 2025-02-15 07:41:30,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44365.25 MB 2025-02-15 07:41:30,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 07:41:30,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31343.34 MB 2025-02-15 07:41:30,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 07:41:30,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:30,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:41:30,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:30,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:41:30,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:41:30,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:41:30,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:41:30,181 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:43:06,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:06,194 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:43:06,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:43:06,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:06,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1132, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:43:06,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:06,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1132, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:43:23,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:43:23,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:43:23,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.43 seconds 2025-02-15 07:43:23,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:23,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20856.66 MB 2025-02-15 07:43:23,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24862.75 MB 2025-02-15 07:43:23,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4006.08 MB 2025-02-15 07:43:23,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52707.72 MB 2025-02-15 07:43:23,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28840.03 MB 2025-02-15 07:43:23,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23867.69 MB 2025-02-15 07:43:23,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33725.42 MB 2025-02-15 07:43:23,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:43:23,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:43:23,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:43:23,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:23,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24862.75 MB 2025-02-15 07:43:23,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21663.81 MB 2025-02-15 07:43:23,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3198.93 MB 2025-02-15 07:43:23,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28840.03 MB 2025-02-15 07:43:23,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38918.95 MB 2025-02-15 07:43:23,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10078.91 MB 2025-02-15 07:43:23,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37039.67 MB 2025-02-15 07:43:25,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:43:25,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:43:25,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:43:25,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:25,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21663.81 MB 2025-02-15 07:43:25,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22194.66 MB 2025-02-15 07:43:25,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:43:25,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38918.95 MB 2025-02-15 07:43:25,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26956.79 MB 2025-02-15 07:43:25,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11962.16 MB 2025-02-15 07:43:25,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26173.20 MB 2025-02-15 07:43:25,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:43:25,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:43:25,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:43:25,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:25,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.66 MB 2025-02-15 07:43:25,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24084.19 MB 2025-02-15 07:43:25,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:43:25,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26956.79 MB 2025-02-15 07:43:25,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27900.51 MB 2025-02-15 07:43:25,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:43:25,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25501.62 MB 2025-02-15 07:43:25,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:43:25,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:43:25,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:43:25,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:25,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24084.19 MB 2025-02-15 07:43:25,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26326.73 MB 2025-02-15 07:43:25,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.54 MB 2025-02-15 07:43:25,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-15 07:43:25,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-15 07:43:25,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:43:25,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.01 MB 2025-02-15 07:43:25,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:43:25,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:43:25,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:43:25,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:25,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.66 MB 2025-02-15 07:43:25,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26326.73 MB 2025-02-15 07:43:25,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.08 MB 2025-02-15 07:43:25,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26956.79 MB 2025-02-15 07:43:25,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-15 07:43:25,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:43:25,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.01 MB 2025-02-15 07:43:26,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:43:26,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:43:26,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:43:26,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:26,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27860.27 MB 2025-02-15 07:43:26,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28627.28 MB 2025-02-15 07:43:26,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:43:26,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33562.82 MB 2025-02-15 07:43:26,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-15 07:43:26,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:43:26,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.06 MB 2025-02-15 07:43:26,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:43:26,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:43:26,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:43:26,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:26,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29040.16 MB 2025-02-15 07:43:26,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29268.28 MB 2025-02-15 07:43:26,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.11 MB 2025-02-15 07:43:26,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-15 07:43:26,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-15 07:43:26,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:43:26,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29499.45 MB 2025-02-15 07:43:26,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:43:26,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:43:26,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.86 seconds 2025-02-15 07:43:26,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:26,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16912.69 MB 2025-02-15 07:43:26,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29468.66 MB 2025-02-15 07:43:26,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12555.98 MB 2025-02-15 07:43:26,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52707.72 MB 2025-02-15 07:43:26,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-15 07:43:26,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18731.76 MB 2025-02-15 07:43:26,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29499.45 MB 2025-02-15 07:43:26,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:43:26,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:43:26,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:43:26,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:26,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29468.66 MB 2025-02-15 07:43:26,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21906.88 MB 2025-02-15 07:43:26,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7561.79 MB 2025-02-15 07:43:26,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-15 07:43:26,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-15 07:43:26,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:43:26,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31972.19 MB 2025-02-15 07:43:26,361 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-15 07:43:26,361 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 07:43:26,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:43:26,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:43:26,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:43:26,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:43:26,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21906.88 MB 2025-02-15 07:43:26,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30316.68 MB 2025-02-15 07:43:26,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-15 07:43:26,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-15 07:43:26,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42337.30 MB 2025-02-15 07:43:26,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-15 07:43:26,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30316.68 MB 2025-02-15 07:43:26,531 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-15 07:43:26,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:26,533 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:43:26,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:26,534 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:43:26,539 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:43:26,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:43:26,540 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:43:26,540 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 07:45:29,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:45:29,778 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:45:29,783 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:45:29,787 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:45:29,787 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:45:29,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:45:29,788 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:46:02,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:46:02,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:46:02,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.22 seconds 2025-02-15 07:46:02,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:02,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-15 07:46:02,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-15 07:46:02,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-15 07:46:02,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54878.27 MB 2025-02-15 07:46:02,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40550.53 MB 2025-02-15 07:46:02,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14327.74 MB 2025-02-15 07:46:02,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43721.65 MB 2025-02-15 07:46:02,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:46:02,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:46:02,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 07:46:02,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:02,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-15 07:46:02,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26586.97 MB 2025-02-15 07:46:02,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8226.00 MB 2025-02-15 07:46:02,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40550.53 MB 2025-02-15 07:46:02,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56971.23 MB 2025-02-15 07:46:02,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16420.70 MB 2025-02-15 07:46:02,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56308.68 MB 2025-02-15 07:46:04,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:46:04,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:46:04,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 07:46:04,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26586.97 MB 2025-02-15 07:46:04,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27117.81 MB 2025-02-15 07:46:04,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:46:04,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56971.23 MB 2025-02-15 07:46:04,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31144.80 MB 2025-02-15 07:46:04,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25826.43 MB 2025-02-15 07:46:04,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31096.51 MB 2025-02-15 07:46:04,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:46:04,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:46:04,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:46:04,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-15 07:46:04,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29007.35 MB 2025-02-15 07:46:04,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:46:04,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31144.80 MB 2025-02-15 07:46:04,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-15 07:46:04,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:46:04,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30424.78 MB 2025-02-15 07:46:04,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:46:04,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:46:04,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:46:04,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.35 MB 2025-02-15 07:46:04,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-15 07:46:04,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:46:04,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-15 07:46:04,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38694.55 MB 2025-02-15 07:46:04,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:46:04,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-15 07:46:04,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:46:04,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:46:04,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:46:04,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-15 07:46:04,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-15 07:46:04,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:46:04,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31144.80 MB 2025-02-15 07:46:04,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38694.55 MB 2025-02-15 07:46:04,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 07:46:04,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-15 07:46:04,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:46:04,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:46:04,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:46:04,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32782.75 MB 2025-02-15 07:46:04,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33549.75 MB 2025-02-15 07:46:04,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:46:04,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38694.55 MB 2025-02-15 07:46:04,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39107.69 MB 2025-02-15 07:46:04,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:46:04,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34257.54 MB 2025-02-15 07:46:04,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:46:04,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:46:04,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:46:04,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33962.64 MB 2025-02-15 07:46:04,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34190.86 MB 2025-02-15 07:46:04,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 07:46:04,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39107.69 MB 2025-02-15 07:46:04,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39107.69 MB 2025-02-15 07:46:04,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:46:04,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34399.61 MB 2025-02-15 07:46:04,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:46:04,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:46:04,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.81 seconds 2025-02-15 07:46:04,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-15 07:46:04,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34391.00 MB 2025-02-15 07:46:04,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14178.89 MB 2025-02-15 07:46:04,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54878.27 MB 2025-02-15 07:46:04,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39107.69 MB 2025-02-15 07:46:04,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15770.58 MB 2025-02-15 07:46:04,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34399.61 MB 2025-02-15 07:46:04,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:46:04,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:46:04,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:46:04,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34391.00 MB 2025-02-15 07:46:04,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25202.74 MB 2025-02-15 07:46:04,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9188.26 MB 2025-02-15 07:46:04,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39107.69 MB 2025-02-15 07:46:04,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39107.69 MB 2025-02-15 07:46:04,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:46:04,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36891.71 MB 2025-02-15 07:46:04,890 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 07:46:04,890 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:46:04,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:46:04,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:46:04,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:46:04,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:46:04,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25202.74 MB 2025-02-15 07:46:04,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33603.60 MB 2025-02-15 07:46:04,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 07:46:04,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39107.69 MB 2025-02-15 07:46:04,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47458.55 MB 2025-02-15 07:46:04,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 07:46:04,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33603.60 MB 2025-02-15 07:46:05,056 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 07:46:05,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:05,057 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:46:05,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:05,058 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:46:05,063 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:46:05,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:05,064 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:46:05,064 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 07:46:55,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:55,362 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:46:55,368 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:46:55,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:55,372 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2192, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:46:55,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:46:55,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2192, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:47:29,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:47:29,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:47:29,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.24 seconds 2025-02-15 07:47:29,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:29,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28242.91 MB 2025-02-15 07:47:29,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36000.28 MB 2025-02-15 07:47:29,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7757.37 MB 2025-02-15 07:47:29,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55809.41 MB 2025-02-15 07:47:29,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40938.50 MB 2025-02-15 07:47:29,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14870.90 MB 2025-02-15 07:47:29,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44962.04 MB 2025-02-15 07:47:29,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:47:29,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:47:29,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 07:47:29,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:29,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36000.28 MB 2025-02-15 07:47:29,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27174.42 MB 2025-02-15 07:47:29,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8825.85 MB 2025-02-15 07:47:29,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40938.50 MB 2025-02-15 07:47:29,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58151.93 MB 2025-02-15 07:47:29,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17213.42 MB 2025-02-15 07:47:29,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58688.98 MB 2025-02-15 07:47:31,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:47:31,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:47:31,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:47:31,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:31,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27174.42 MB 2025-02-15 07:47:31,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27705.27 MB 2025-02-15 07:47:31,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:47:31,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58151.93 MB 2025-02-15 07:47:31,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31130.12 MB 2025-02-15 07:47:31,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27021.80 MB 2025-02-15 07:47:31,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31683.81 MB 2025-02-15 07:47:31,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:47:31,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:47:31,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:47:31,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:31,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27705.27 MB 2025-02-15 07:47:31,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29594.80 MB 2025-02-15 07:47:31,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:47:31,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31130.12 MB 2025-02-15 07:47:31,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33961.28 MB 2025-02-15 07:47:31,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:47:31,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31012.23 MB 2025-02-15 07:47:31,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:47:31,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:47:31,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:47:31,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:31,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29594.80 MB 2025-02-15 07:47:31,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31836.66 MB 2025-02-15 07:47:31,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:47:31,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33961.28 MB 2025-02-15 07:47:31,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39623.59 MB 2025-02-15 07:47:31,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:47:31,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37380.94 MB 2025-02-15 07:47:31,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:47:31,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:47:31,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:47:31,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:31,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27705.27 MB 2025-02-15 07:47:31,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31836.66 MB 2025-02-15 07:47:31,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:47:31,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31130.12 MB 2025-02-15 07:47:31,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39623.59 MB 2025-02-15 07:47:31,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 07:47:31,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37380.94 MB 2025-02-15 07:47:32,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:47:32,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:47:32,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:47:32,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:32,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33370.20 MB 2025-02-15 07:47:32,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34137.20 MB 2025-02-15 07:47:32,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:47:32,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39623.59 MB 2025-02-15 07:47:32,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 07:47:32,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:47:32,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34844.99 MB 2025-02-15 07:47:32,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:47:32,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:47:32,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:47:32,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:32,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34550.09 MB 2025-02-15 07:47:32,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34778.85 MB 2025-02-15 07:47:32,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.76 MB 2025-02-15 07:47:32,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 07:47:32,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 07:47:32,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:47:32,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35013.20 MB 2025-02-15 07:47:32,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:47:32,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:47:32,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.87 seconds 2025-02-15 07:47:32,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:32,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20605.81 MB 2025-02-15 07:47:32,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34979.70 MB 2025-02-15 07:47:32,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14373.89 MB 2025-02-15 07:47:32,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55809.41 MB 2025-02-15 07:47:32,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 07:47:32,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15770.58 MB 2025-02-15 07:47:32,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35013.20 MB 2025-02-15 07:47:32,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:47:32,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:47:32,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 07:47:32,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:32,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34979.70 MB 2025-02-15 07:47:32,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25604.28 MB 2025-02-15 07:47:32,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9375.43 MB 2025-02-15 07:47:32,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 07:47:32,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 07:47:32,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:47:32,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37486.46 MB 2025-02-15 07:47:32,565 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 07:47:32,566 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:47:32,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:47:32,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:47:32,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:47:32,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:47:32,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25604.28 MB 2025-02-15 07:47:32,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34026.60 MB 2025-02-15 07:47:32,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 07:47:32,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 07:47:32,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48412.75 MB 2025-02-15 07:47:32,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 07:47:32,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34026.60 MB 2025-02-15 07:47:32,831 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 07:47:32,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:32,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:47:32,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:32,834 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:47:32,840 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:47:32,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:32,841 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:47:32,841 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:47:42,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:42,360 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:47:42,365 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:47:42,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:42,368 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:47:42,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:47:42,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:48:03,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:48:03,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:48:03,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.02 seconds 2025-02-15 07:48:03,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:03,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-15 07:48:03,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-15 07:48:03,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-15 07:48:03,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60972.60 MB 2025-02-15 07:48:03,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37937.48 MB 2025-02-15 07:48:03,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23035.12 MB 2025-02-15 07:48:03,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-15 07:48:03,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:48:03,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:48:03,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:48:03,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:03,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-15 07:48:03,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-15 07:48:03,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-15 07:48:03,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37937.48 MB 2025-02-15 07:48:03,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 07:48:03,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3380.61 MB 2025-02-15 07:48:03,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37980.92 MB 2025-02-15 07:48:05,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:48:05,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:48:05,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:48:05,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-15 07:48:05,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-15 07:48:05,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:48:05,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 07:48:05,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33193.72 MB 2025-02-15 07:48:05,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8124.37 MB 2025-02-15 07:48:05,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27253.48 MB 2025-02-15 07:48:05,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:48:05,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:48:05,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:48:05,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-15 07:48:05,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25165.37 MB 2025-02-15 07:48:05,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1890.44 MB 2025-02-15 07:48:05,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33193.72 MB 2025-02-15 07:48:05,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33193.72 MB 2025-02-15 07:48:05,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:48:05,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26582.80 MB 2025-02-15 07:48:05,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:48:05,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:48:05,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:48:05,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25165.37 MB 2025-02-15 07:48:05,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27407.23 MB 2025-02-15 07:48:05,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:48:05,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33193.72 MB 2025-02-15 07:48:05,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35553.02 MB 2025-02-15 07:48:05,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 07:48:05,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32951.51 MB 2025-02-15 07:48:05,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:48:05,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:48:05,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:48:05,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-15 07:48:05,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27407.23 MB 2025-02-15 07:48:05,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.29 MB 2025-02-15 07:48:05,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33193.72 MB 2025-02-15 07:48:05,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35553.02 MB 2025-02-15 07:48:05,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 07:48:05,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32951.51 MB 2025-02-15 07:48:05,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:48:05,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:48:05,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:48:05,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28940.77 MB 2025-02-15 07:48:05,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29707.77 MB 2025-02-15 07:48:05,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:48:05,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35553.02 MB 2025-02-15 07:48:05,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-15 07:48:05,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:48:05,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30415.56 MB 2025-02-15 07:48:05,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:48:05,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:48:05,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:48:05,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30120.66 MB 2025-02-15 07:48:05,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30347.09 MB 2025-02-15 07:48:05,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.43 MB 2025-02-15 07:48:05,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35966.16 MB 2025-02-15 07:48:05,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-15 07:48:05,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:48:05,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30587.46 MB 2025-02-15 07:48:05,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:48:05,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:48:05,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.42 seconds 2025-02-15 07:48:05,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:05,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-15 07:48:05,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30547.13 MB 2025-02-15 07:48:05,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12909.76 MB 2025-02-15 07:48:05,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60972.60 MB 2025-02-15 07:48:05,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-15 07:48:05,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25006.44 MB 2025-02-15 07:48:05,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30587.46 MB 2025-02-15 07:48:06,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:48:06,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:48:06,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:48:06,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:06,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30547.13 MB 2025-02-15 07:48:06,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22626.57 MB 2025-02-15 07:48:06,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7920.56 MB 2025-02-15 07:48:06,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35966.16 MB 2025-02-15 07:48:06,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-15 07:48:06,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:48:06,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33045.90 MB 2025-02-15 07:48:06,082 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 07:48:06,082 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:48:06,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:48:06,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:48:06,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:48:06,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:48:06,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22626.57 MB 2025-02-15 07:48:06,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31023.22 MB 2025-02-15 07:48:06,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-15 07:48:06,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35966.16 MB 2025-02-15 07:48:06,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44312.82 MB 2025-02-15 07:48:06,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 07:48:06,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31023.22 MB 2025-02-15 07:48:06,245 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 07:48:06,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:06,247 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:48:06,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:06,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:48:06,252 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:48:06,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:06,253 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:48:06,254 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:48:59,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:59,428 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:48:59,433 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:48:59,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:59,437 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 149, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:48:59,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:48:59,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 149, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:49:01,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:49:01,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:49:01,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.33 seconds 2025-02-15 07:49:01,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:01,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14006.96 MB 2025-02-15 07:49:01,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14534.26 MB 2025-02-15 07:49:01,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 527.30 MB 2025-02-15 07:49:01,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52659.49 MB 2025-02-15 07:49:01,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17425.24 MB 2025-02-15 07:49:01,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35234.25 MB 2025-02-15 07:49:01,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23478.33 MB 2025-02-15 07:49:01,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:49:01,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:49:01,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:49:01,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:01,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14534.26 MB 2025-02-15 07:49:01,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14762.31 MB 2025-02-15 07:49:01,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.04 MB 2025-02-15 07:49:01,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17425.24 MB 2025-02-15 07:49:01,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17939.04 MB 2025-02-15 07:49:01,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 513.80 MB 2025-02-15 07:49:01,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16571.66 MB 2025-02-15 07:49:02,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:49:02,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:49:02,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 07:49:02,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14762.31 MB 2025-02-15 07:49:02,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14954.74 MB 2025-02-15 07:49:02,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 07:49:02,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17939.04 MB 2025-02-15 07:49:02,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17939.04 MB 2025-02-15 07:49:02,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:49:02,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18932.99 MB 2025-02-15 07:49:02,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:49:02,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:49:02,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:49:02,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.67 MB 2025-02-15 07:49:02,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15639.46 MB 2025-02-15 07:49:02,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 07:49:02,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17939.04 MB 2025-02-15 07:49:02,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17939.04 MB 2025-02-15 07:49:02,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:49:02,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16153.28 MB 2025-02-15 07:49:02,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:49:02,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:49:02,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:49:02,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15639.46 MB 2025-02-15 07:49:02,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16452.17 MB 2025-02-15 07:49:02,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 07:49:02,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17939.04 MB 2025-02-15 07:49:02,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19314.77 MB 2025-02-15 07:49:02,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 07:49:02,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.94 MB 2025-02-15 07:49:02,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:49:02,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:49:02,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:49:02,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.67 MB 2025-02-15 07:49:02,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16452.17 MB 2025-02-15 07:49:02,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 07:49:02,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17939.04 MB 2025-02-15 07:49:02,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19314.77 MB 2025-02-15 07:49:02,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 07:49:02,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.94 MB 2025-02-15 07:49:02,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:49:02,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:49:02,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:49:02,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17008.08 MB 2025-02-15 07:49:02,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17286.12 MB 2025-02-15 07:49:02,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 07:49:02,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19314.77 MB 2025-02-15 07:49:02,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19459.47 MB 2025-02-15 07:49:02,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-15 07:49:02,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17552.70 MB 2025-02-15 07:49:02,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:49:02,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:49:02,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:49:02,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17435.80 MB 2025-02-15 07:49:02,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17664.82 MB 2025-02-15 07:49:02,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-15 07:49:02,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19459.47 MB 2025-02-15 07:49:02,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19459.47 MB 2025-02-15 07:49:02,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:49:02,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17664.82 MB 2025-02-15 07:49:02,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:49:02,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:49:02,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-15 07:49:02,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13487.83 MB 2025-02-15 07:49:02,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17865.84 MB 2025-02-15 07:49:02,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4378.01 MB 2025-02-15 07:49:02,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52659.49 MB 2025-02-15 07:49:02,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19459.47 MB 2025-02-15 07:49:02,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33200.01 MB 2025-02-15 07:49:02,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17865.84 MB 2025-02-15 07:49:02,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:49:02,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:49:02,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.33 seconds 2025-02-15 07:49:02,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:02,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17865.84 MB 2025-02-15 07:49:02,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17288.04 MB 2025-02-15 07:49:02,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -577.80 MB 2025-02-15 07:49:02,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19459.47 MB 2025-02-15 07:49:02,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19593.69 MB 2025-02-15 07:49:02,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-15 07:49:02,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18971.00 MB 2025-02-15 07:49:02,998 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 07:49:02,998 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-15 07:49:03,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:49:03,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:49:03,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:49:03,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:03,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17288.04 MB 2025-02-15 07:49:03,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25725.51 MB 2025-02-15 07:49:03,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 07:49:03,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19593.69 MB 2025-02-15 07:49:03,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30079.45 MB 2025-02-15 07:49:03,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 07:49:03,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25725.51 MB 2025-02-15 07:49:03,163 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 07:49:03,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:03,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:49:03,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:03,165 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:49:03,170 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:49:03,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:03,171 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:49:03,171 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-15 07:49:12,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:12,308 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:49:12,313 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:49:12,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:12,316 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1133, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:49:12,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:12,317 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1133, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:49:30,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:49:30,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:49:30,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.71 seconds 2025-02-15 07:49:30,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:30,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20863.63 MB 2025-02-15 07:49:30,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24873.39 MB 2025-02-15 07:49:30,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4009.75 MB 2025-02-15 07:49:30,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38468.06 MB 2025-02-15 07:49:30,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26778.53 MB 2025-02-15 07:49:30,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11689.53 MB 2025-02-15 07:49:30,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33733.20 MB 2025-02-15 07:49:30,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:49:30,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:49:30,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 07:49:30,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:30,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24873.39 MB 2025-02-15 07:49:30,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21669.01 MB 2025-02-15 07:49:30,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3204.37 MB 2025-02-15 07:49:30,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26778.53 MB 2025-02-15 07:49:30,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35766.93 MB 2025-02-15 07:49:30,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8988.39 MB 2025-02-15 07:49:30,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35961.82 MB 2025-02-15 07:49:32,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:49:32,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:49:32,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.05 seconds 2025-02-15 07:49:32,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21669.01 MB 2025-02-15 07:49:32,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22199.85 MB 2025-02-15 07:49:32,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:49:32,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35766.93 MB 2025-02-15 07:49:32,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24893.19 MB 2025-02-15 07:49:32,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10873.73 MB 2025-02-15 07:49:32,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26179.44 MB 2025-02-15 07:49:32,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:49:32,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:49:32,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:49:32,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22199.85 MB 2025-02-15 07:49:32,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24089.39 MB 2025-02-15 07:49:32,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:49:32,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 07:49:32,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27724.35 MB 2025-02-15 07:49:32,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 07:49:32,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25506.82 MB 2025-02-15 07:49:32,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:49:32,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:49:32,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 07:49:32,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24089.39 MB 2025-02-15 07:49:32,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26331.24 MB 2025-02-15 07:49:32,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:49:32,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27724.35 MB 2025-02-15 07:49:32,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33386.66 MB 2025-02-15 07:49:32,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:49:32,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31875.53 MB 2025-02-15 07:49:32,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:49:32,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:49:32,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:49:32,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22199.85 MB 2025-02-15 07:49:32,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26331.24 MB 2025-02-15 07:49:32,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:49:32,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 07:49:32,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33386.66 MB 2025-02-15 07:49:32,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 07:49:32,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31875.53 MB 2025-02-15 07:49:32,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:49:32,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:49:32,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:49:32,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27864.79 MB 2025-02-15 07:49:32,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28631.79 MB 2025-02-15 07:49:32,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:49:32,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33386.66 MB 2025-02-15 07:49:32,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 07:49:32,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 07:49:32,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29339.58 MB 2025-02-15 07:49:32,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:49:32,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:49:32,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:49:32,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29044.68 MB 2025-02-15 07:49:32,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29272.66 MB 2025-02-15 07:49:32,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.99 MB 2025-02-15 07:49:32,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 07:49:32,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 07:49:32,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:49:32,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29455.02 MB 2025-02-15 07:49:32,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:49:32,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:49:32,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.27 seconds 2025-02-15 07:49:32,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16916.17 MB 2025-02-15 07:49:32,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29473.51 MB 2025-02-15 07:49:32,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12557.35 MB 2025-02-15 07:49:32,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38468.06 MB 2025-02-15 07:49:32,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 07:49:32,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4664.07 MB 2025-02-15 07:49:32,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29473.51 MB 2025-02-15 07:49:32,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:49:32,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:49:32,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:49:32,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29473.51 MB 2025-02-15 07:49:32,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21905.73 MB 2025-02-15 07:49:32,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7567.79 MB 2025-02-15 07:49:32,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 07:49:32,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 07:49:32,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:49:32,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31972.59 MB 2025-02-15 07:49:32,879 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 07:49:32,879 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:49:32,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:49:32,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:49:32,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:49:32,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:49:32,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21905.73 MB 2025-02-15 07:49:32,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30302.49 MB 2025-02-15 07:49:32,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-15 07:49:32,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 07:49:32,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42152.76 MB 2025-02-15 07:49:32,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-15 07:49:32,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30302.49 MB 2025-02-15 07:49:33,043 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 07:49:33,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:33,045 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:49:33,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:33,046 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:49:33,050 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:49:33,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:49:33,052 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:49:33,052 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:50:04,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:04,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:50:04,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:50:04,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:04,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:50:04,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:04,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:50:06,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:50:06,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:50:06,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-15 07:50:06,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:06,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-15 07:50:06,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-15 07:50:06,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-15 07:50:06,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54674.85 MB 2025-02-15 07:50:06,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 07:50:06,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36310.09 MB 2025-02-15 07:50:06,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23499.24 MB 2025-02-15 07:50:06,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:50:06,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:50:06,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:50:06,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:06,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-15 07:50:06,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.27 MB 2025-02-15 07:50:06,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.48 MB 2025-02-15 07:50:06,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 07:50:06,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 07:50:06,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:50:06,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16616.57 MB 2025-02-15 07:50:07,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:50:07,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:50:07,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-15 07:50:07,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.27 MB 2025-02-15 07:50:07,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14978.03 MB 2025-02-15 07:50:07,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-15 07:50:07,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 07:50:07,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 07:50:07,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 07:50:07,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18954.96 MB 2025-02-15 07:50:07,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:50:07,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:50:07,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:50:07,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-15 07:50:07,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15667.47 MB 2025-02-15 07:50:07,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-15 07:50:07,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 07:50:07,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 07:50:07,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:50:07,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16184.84 MB 2025-02-15 07:50:07,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:50:07,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:50:07,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 07:50:07,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15667.47 MB 2025-02-15 07:50:07,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-15 07:50:07,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-15 07:50:07,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 07:50:07,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19625.15 MB 2025-02-15 07:50:07,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1732.25 MB 2025-02-15 07:50:07,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-15 07:50:07,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:50:07,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:50:07,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 07:50:07,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-15 07:50:07,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-15 07:50:07,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-15 07:50:07,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 07:50:07,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19625.15 MB 2025-02-15 07:50:07,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1732.25 MB 2025-02-15 07:50:07,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-15 07:50:07,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:50:07,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:50:07,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:50:07,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17045.54 MB 2025-02-15 07:50:07,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.49 MB 2025-02-15 07:50:07,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-15 07:50:07,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19625.15 MB 2025-02-15 07:50:07,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19776.14 MB 2025-02-15 07:50:07,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 07:50:07,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17596.29 MB 2025-02-15 07:50:07,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:50:07,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:50:07,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:50:07,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.20 MB 2025-02-15 07:50:07,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17680.94 MB 2025-02-15 07:50:07,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.74 MB 2025-02-15 07:50:07,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19776.14 MB 2025-02-15 07:50:07,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19780.34 MB 2025-02-15 07:50:07,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 07:50:07,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17696.48 MB 2025-02-15 07:50:07,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:50:07,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:50:07,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-15 07:50:07,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:07,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-15 07:50:07,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17881.86 MB 2025-02-15 07:50:07,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4383.58 MB 2025-02-15 07:50:07,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54674.85 MB 2025-02-15 07:50:07,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19780.34 MB 2025-02-15 07:50:07,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34894.51 MB 2025-02-15 07:50:07,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17881.86 MB 2025-02-15 07:50:08,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:50:08,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:50:08,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 07:50:08,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:08,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17881.86 MB 2025-02-15 07:50:08,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17301.43 MB 2025-02-15 07:50:08,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -580.44 MB 2025-02-15 07:50:08,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19780.34 MB 2025-02-15 07:50:08,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19780.34 MB 2025-02-15 07:50:08,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:50:08,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18986.19 MB 2025-02-15 07:50:08,208 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 07:50:08,208 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:50:08,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:50:08,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:50:08,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:50:08,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:50:08,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17301.43 MB 2025-02-15 07:50:08,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25734.73 MB 2025-02-15 07:50:08,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 07:50:08,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19780.34 MB 2025-02-15 07:50:08,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30261.90 MB 2025-02-15 07:50:08,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 07:50:08,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25734.73 MB 2025-02-15 07:50:08,371 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 07:50:08,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:08,373 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:50:08,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:08,374 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:50:08,378 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:50:08,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:50:08,379 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:50:08,380 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:51:01,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:01,367 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:51:01,371 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:51:01,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:01,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 674, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:51:01,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:01,376 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 674, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:51:11,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:51:11,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:51:11,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.39 seconds 2025-02-15 07:51:11,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:11,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17665.25 MB 2025-02-15 07:51:11,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20050.49 MB 2025-02-15 07:51:11,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.25 MB 2025-02-15 07:51:11,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38646.32 MB 2025-02-15 07:51:11,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25151.14 MB 2025-02-15 07:51:11,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13495.17 MB 2025-02-15 07:51:11,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.56 MB 2025-02-15 07:51:11,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:51:11,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:51:11,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 07:51:11,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:11,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20050.49 MB 2025-02-15 07:51:11,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19282.81 MB 2025-02-15 07:51:11,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -767.68 MB 2025-02-15 07:51:11,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25151.14 MB 2025-02-15 07:51:11,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31637.64 MB 2025-02-15 07:51:11,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6486.49 MB 2025-02-15 07:51:11,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28638.40 MB 2025-02-15 07:51:13,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:51:13,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:51:13,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:51:13,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:13,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19282.81 MB 2025-02-15 07:51:13,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19813.66 MB 2025-02-15 07:51:13,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:51:13,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31637.64 MB 2025-02-15 07:51:13,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24889.00 MB 2025-02-15 07:51:13,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6748.64 MB 2025-02-15 07:51:13,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23792.20 MB 2025-02-15 07:51:13,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:51:13,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:51:13,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 07:51:13,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:13,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19813.66 MB 2025-02-15 07:51:13,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21703.19 MB 2025-02-15 07:51:13,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:51:13,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24889.00 MB 2025-02-15 07:51:13,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25832.72 MB 2025-02-15 07:51:13,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:51:13,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23120.62 MB 2025-02-15 07:51:13,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:51:13,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:51:13,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:51:13,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:13,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21703.19 MB 2025-02-15 07:51:13,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23945.05 MB 2025-02-15 07:51:13,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:51:13,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25832.72 MB 2025-02-15 07:51:13,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-15 07:51:13,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 07:51:13,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29489.33 MB 2025-02-15 07:51:13,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:51:13,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:51:13,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:51:13,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:13,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19813.66 MB 2025-02-15 07:51:13,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23945.05 MB 2025-02-15 07:51:13,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:51:13,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24889.00 MB 2025-02-15 07:51:13,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-15 07:51:13,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7079.99 MB 2025-02-15 07:51:13,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29489.33 MB 2025-02-15 07:51:14,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:51:14,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:51:14,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:51:14,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:14,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25478.59 MB 2025-02-15 07:51:14,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26245.59 MB 2025-02-15 07:51:14,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:51:14,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31968.99 MB 2025-02-15 07:51:14,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 07:51:14,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 07:51:14,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26953.38 MB 2025-02-15 07:51:14,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:51:14,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:51:14,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:51:14,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:14,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26658.48 MB 2025-02-15 07:51:14,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26887.59 MB 2025-02-15 07:51:14,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-15 07:51:14,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 07:51:14,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 07:51:14,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:51:14,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27090.50 MB 2025-02-15 07:51:14,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:51:14,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:51:14,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.80 seconds 2025-02-15 07:51:14,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:14,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15316.98 MB 2025-02-15 07:51:14,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27088.66 MB 2025-02-15 07:51:14,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11771.68 MB 2025-02-15 07:51:14,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38646.32 MB 2025-02-15 07:51:14,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 07:51:14,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6260.00 MB 2025-02-15 07:51:14,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27090.50 MB 2025-02-15 07:51:14,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:51:14,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:51:14,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:51:14,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:14,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27088.66 MB 2025-02-15 07:51:14,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20321.37 MB 2025-02-15 07:51:14,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6767.30 MB 2025-02-15 07:51:14,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 07:51:14,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-15 07:51:14,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:51:14,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29600.33 MB 2025-02-15 07:51:14,464 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:51:14,465 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:51:14,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:51:14,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:51:14,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:51:14,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:51:14,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20321.37 MB 2025-02-15 07:51:14,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28760.39 MB 2025-02-15 07:51:14,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:51:14,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-15 07:51:14,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40777.02 MB 2025-02-15 07:51:14,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:51:14,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28760.39 MB 2025-02-15 07:51:14,629 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:51:14,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:14,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:51:14,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:14,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:51:14,636 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:51:14,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:51:14,637 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:51:14,637 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:52:03,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:03,906 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:52:03,911 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:52:03,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:03,915 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:52:03,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:03,916 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:52:23,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:52:23,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:52:23,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.54 seconds 2025-02-15 07:52:23,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:23,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-15 07:52:23,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-15 07:52:23,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-15 07:52:23,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53362.03 MB 2025-02-15 07:52:23,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37704.70 MB 2025-02-15 07:52:23,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15657.34 MB 2025-02-15 07:52:23,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-15 07:52:23,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:52:23,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:52:23,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:52:23,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:23,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-15 07:52:23,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-15 07:52:23,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-15 07:52:23,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37704.70 MB 2025-02-15 07:52:23,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-15 07:52:23,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8831.11 MB 2025-02-15 07:52:23,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39454.03 MB 2025-02-15 07:52:25,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:52:25,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:52:25,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:52:25,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-15 07:52:25,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-15 07:52:25,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:52:25,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46535.80 MB 2025-02-15 07:52:25,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 07:52:25,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17475.57 MB 2025-02-15 07:52:25,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.99 MB 2025-02-15 07:52:25,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:52:25,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:52:25,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:52:25,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-15 07:52:25,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-15 07:52:25,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:52:25,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 07:52:25,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 07:52:25,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:52:25,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-15 07:52:25,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:52:25,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:52:25,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 07:52:25,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-15 07:52:25,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-15 07:52:25,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:52:25,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 07:52:25,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34722.55 MB 2025-02-15 07:52:25,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:52:25,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-15 07:52:25,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:52:25,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:52:25,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 07:52:25,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-15 07:52:25,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-15 07:52:25,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:52:25,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 07:52:25,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34722.55 MB 2025-02-15 07:52:25,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:52:25,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-15 07:52:25,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:52:25,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:52:25,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:52:25,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-15 07:52:25,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-15 07:52:25,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:52:25,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34722.55 MB 2025-02-15 07:52:25,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35137.78 MB 2025-02-15 07:52:25,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:52:25,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-15 07:52:25,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:52:25,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:52:25,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:52:25,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-15 07:52:25,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29916.71 MB 2025-02-15 07:52:25,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 07:52:25,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35137.78 MB 2025-02-15 07:52:25,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35137.78 MB 2025-02-15 07:52:25,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:52:25,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30148.70 MB 2025-02-15 07:52:25,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:52:25,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:52:25,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.04 seconds 2025-02-15 07:52:25,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:25,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-15 07:52:25,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30117.56 MB 2025-02-15 07:52:25,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12769.37 MB 2025-02-15 07:52:25,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53362.03 MB 2025-02-15 07:52:25,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35137.78 MB 2025-02-15 07:52:25,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18224.25 MB 2025-02-15 07:52:25,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30148.70 MB 2025-02-15 07:52:26,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:52:26,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:52:26,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:52:26,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:26,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30117.56 MB 2025-02-15 07:52:26,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22342.03 MB 2025-02-15 07:52:26,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7775.53 MB 2025-02-15 07:52:26,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35137.78 MB 2025-02-15 07:52:26,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35137.78 MB 2025-02-15 07:52:26,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:52:26,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32620.32 MB 2025-02-15 07:52:26,247 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 07:52:26,248 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 07:52:26,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:52:26,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:52:26,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:52:26,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:52:26,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22342.03 MB 2025-02-15 07:52:26,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30750.80 MB 2025-02-15 07:52:26,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.77 MB 2025-02-15 07:52:26,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35137.78 MB 2025-02-15 07:52:26,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39317.41 MB 2025-02-15 07:52:26,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 07:52:26,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30750.80 MB 2025-02-15 07:52:26,415 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 07:52:26,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:26,417 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:52:26,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:26,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:52:26,422 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:52:26,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:52:26,424 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:52:26,424 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 07:53:59,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:53:59,924 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:53:59,929 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:53:59,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:53:59,933 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1246, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:53:59,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:53:59,934 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1246, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:54:19,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:54:19,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:54:19,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.09 seconds 2025-02-15 07:54:19,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:19,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21651.03 MB 2025-02-15 07:54:19,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26061.34 MB 2025-02-15 07:54:19,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4410.31 MB 2025-02-15 07:54:19,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47676.65 MB 2025-02-15 07:54:19,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37620.81 MB 2025-02-15 07:54:19,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10055.84 MB 2025-02-15 07:54:19,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34972.78 MB 2025-02-15 07:54:19,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:54:19,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:54:19,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 07:54:19,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:19,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26061.34 MB 2025-02-15 07:54:19,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22255.42 MB 2025-02-15 07:54:19,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3805.93 MB 2025-02-15 07:54:19,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37620.81 MB 2025-02-15 07:54:19,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46374.32 MB 2025-02-15 07:54:19,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8753.51 MB 2025-02-15 07:54:19,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39222.98 MB 2025-02-15 07:54:21,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:54:21,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:54:21,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 07:54:21,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22255.42 MB 2025-02-15 07:54:21,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22786.26 MB 2025-02-15 07:54:21,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:54:21,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46374.32 MB 2025-02-15 07:54:21,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29030.88 MB 2025-02-15 07:54:21,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17343.45 MB 2025-02-15 07:54:21,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26764.81 MB 2025-02-15 07:54:21,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:54:21,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:54:21,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:54:21,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22786.26 MB 2025-02-15 07:54:21,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24675.79 MB 2025-02-15 07:54:21,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:54:21,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29030.88 MB 2025-02-15 07:54:21,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29030.88 MB 2025-02-15 07:54:21,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:54:21,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26093.22 MB 2025-02-15 07:54:21,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:54:21,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:54:21,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:54:21,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24675.79 MB 2025-02-15 07:54:21,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26917.65 MB 2025-02-15 07:54:21,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:54:21,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29030.88 MB 2025-02-15 07:54:21,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 07:54:21,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:54:21,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32461.93 MB 2025-02-15 07:54:21,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:54:21,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:54:21,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:54:21,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22786.26 MB 2025-02-15 07:54:21,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26917.65 MB 2025-02-15 07:54:21,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:54:21,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29030.88 MB 2025-02-15 07:54:21,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 07:54:21,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:54:21,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32461.93 MB 2025-02-15 07:54:21,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:54:21,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:54:21,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 07:54:21,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28451.19 MB 2025-02-15 07:54:21,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29218.19 MB 2025-02-15 07:54:21,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:54:21,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34693.19 MB 2025-02-15 07:54:21,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35108.42 MB 2025-02-15 07:54:21,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:54:21,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29925.98 MB 2025-02-15 07:54:21,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:54:21,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:54:21,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:54:21,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29631.08 MB 2025-02-15 07:54:21,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29859.65 MB 2025-02-15 07:54:21,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-15 07:54:21,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35108.42 MB 2025-02-15 07:54:21,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35108.42 MB 2025-02-15 07:54:21,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:54:21,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30092.33 MB 2025-02-15 07:54:21,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:54:21,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:54:21,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.50 seconds 2025-02-15 07:54:21,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17309.87 MB 2025-02-15 07:54:21,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30060.50 MB 2025-02-15 07:54:21,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12750.63 MB 2025-02-15 07:54:21,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47676.65 MB 2025-02-15 07:54:21,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35108.42 MB 2025-02-15 07:54:21,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12568.23 MB 2025-02-15 07:54:21,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30092.33 MB 2025-02-15 07:54:21,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:54:21,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:54:21,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 07:54:21,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30060.50 MB 2025-02-15 07:54:21,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.49 MB 2025-02-15 07:54:21,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7755.01 MB 2025-02-15 07:54:21,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35108.42 MB 2025-02-15 07:54:21,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35108.42 MB 2025-02-15 07:54:21,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:54:21,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32564.79 MB 2025-02-15 07:54:21,717 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 07:54:21,717 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 07:54:21,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:54:21,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:54:21,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:54:21,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:54:21,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22305.49 MB 2025-02-15 07:54:21,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30719.40 MB 2025-02-15 07:54:21,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.92 MB 2025-02-15 07:54:21,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35108.42 MB 2025-02-15 07:54:21,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39292.24 MB 2025-02-15 07:54:21,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 07:54:21,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30719.40 MB 2025-02-15 07:54:21,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 07:54:21,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:54:21,885 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:54:21,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:54:21,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:54:21,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:54:21,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:54:21,892 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:54:21,892 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 07:55:19,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:19,726 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:55:19,734 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:55:19,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:19,742 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2041, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:55:19,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:19,744 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2041, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:55:51,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:55:51,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:55:51,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.80 seconds 2025-02-15 07:55:51,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:51,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27190.72 MB 2025-02-15 07:55:51,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34413.71 MB 2025-02-15 07:55:51,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7222.98 MB 2025-02-15 07:55:51,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 07:55:51,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40424.70 MB 2025-02-15 07:55:51,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7230.98 MB 2025-02-15 07:55:51,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43230.37 MB 2025-02-15 07:55:51,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:55:51,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:55:51,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:55:51,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:51,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34413.71 MB 2025-02-15 07:55:51,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26389.42 MB 2025-02-15 07:55:51,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8024.28 MB 2025-02-15 07:55:51,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40424.70 MB 2025-02-15 07:55:51,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54890.86 MB 2025-02-15 07:55:51,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14466.15 MB 2025-02-15 07:55:51,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52978.89 MB 2025-02-15 07:55:53,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:55:53,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:55:53,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 07:55:53,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:53,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26389.42 MB 2025-02-15 07:55:53,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26920.26 MB 2025-02-15 07:55:53,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:55:53,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54890.86 MB 2025-02-15 07:55:53,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31144.80 MB 2025-02-15 07:55:53,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23746.05 MB 2025-02-15 07:55:53,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30898.96 MB 2025-02-15 07:55:53,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:55:53,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:55:53,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:55:53,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:53,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26920.26 MB 2025-02-15 07:55:53,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28809.80 MB 2025-02-15 07:55:53,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:55:53,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31144.80 MB 2025-02-15 07:55:53,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-15 07:55:53,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:55:53,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30227.23 MB 2025-02-15 07:55:53,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:55:53,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:55:53,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:55:53,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:53,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28809.80 MB 2025-02-15 07:55:53,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31051.65 MB 2025-02-15 07:55:53,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:55:53,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-15 07:55:53,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38694.55 MB 2025-02-15 07:55:53,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 07:55:53,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36595.94 MB 2025-02-15 07:55:53,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:55:53,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:55:53,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 07:55:53,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:53,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26920.26 MB 2025-02-15 07:55:53,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31051.65 MB 2025-02-15 07:55:53,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:55:53,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31144.80 MB 2025-02-15 07:55:53,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38694.55 MB 2025-02-15 07:55:53,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 07:55:53,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36595.94 MB 2025-02-15 07:55:54,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:55:54,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:55:54,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:55:54,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:54,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32585.20 MB 2025-02-15 07:55:54,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33352.20 MB 2025-02-15 07:55:54,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:55:54,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38694.55 MB 2025-02-15 07:55:54,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-15 07:55:54,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 07:55:54,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34059.99 MB 2025-02-15 07:55:54,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:55:54,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:55:54,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:55:54,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:54,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33765.09 MB 2025-02-15 07:55:54,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33993.10 MB 2025-02-15 07:55:54,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.01 MB 2025-02-15 07:55:54,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-15 07:55:54,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-15 07:55:54,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:55:54,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34213.28 MB 2025-02-15 07:55:54,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:55:54,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:55:54,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.33 seconds 2025-02-15 07:55:54,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:54,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20079.71 MB 2025-02-15 07:55:54,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34193.73 MB 2025-02-15 07:55:54,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14114.02 MB 2025-02-15 07:55:54,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47655.68 MB 2025-02-15 07:55:54,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-15 07:55:54,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8543.80 MB 2025-02-15 07:55:54,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34213.28 MB 2025-02-15 07:55:54,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:55:54,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:55:54,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:55:54,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:54,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34193.73 MB 2025-02-15 07:55:54,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25077.47 MB 2025-02-15 07:55:54,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9116.26 MB 2025-02-15 07:55:54,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-15 07:55:54,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-15 07:55:54,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:55:54,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36699.87 MB 2025-02-15 07:55:54,374 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 07:55:54,375 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:55:54,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:55:54,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:55:54,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:55:54,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:55:54,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25077.47 MB 2025-02-15 07:55:54,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33498.24 MB 2025-02-15 07:55:54,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 07:55:54,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-15 07:55:54,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47483.72 MB 2025-02-15 07:55:54,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 07:55:54,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33498.24 MB 2025-02-15 07:55:54,542 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 07:55:54,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:54,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:55:54,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:54,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:55:54,549 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:55:54,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:55:54,550 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:55:54,550 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:56:50,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:56:50,736 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:56:50,741 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:56:50,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:56:50,745 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:56:50,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:56:50,746 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:57:11,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:57:11,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:57:11,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.54 seconds 2025-02-15 07:57:11,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:11,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.55 MB 2025-02-15 07:57:11,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26880.11 MB 2025-02-15 07:57:11,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4685.56 MB 2025-02-15 07:57:11,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55855.55 MB 2025-02-15 07:57:11,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37895.54 MB 2025-02-15 07:57:11,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17960.01 MB 2025-02-15 07:57:11,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35742.79 MB 2025-02-15 07:57:11,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:57:11,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:57:11,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:57:11,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:11,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26880.11 MB 2025-02-15 07:57:11,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22660.91 MB 2025-02-15 07:57:11,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4219.20 MB 2025-02-15 07:57:11,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37895.54 MB 2025-02-15 07:57:11,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37895.54 MB 2025-02-15 07:57:11,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:57:11,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34188.01 MB 2025-02-15 07:57:13,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:57:13,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:57:13,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 07:57:13,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22660.91 MB 2025-02-15 07:57:13,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23191.76 MB 2025-02-15 07:57:13,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:57:13,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37895.54 MB 2025-02-15 07:57:13,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 07:57:13,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8868.86 MB 2025-02-15 07:57:13,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27170.30 MB 2025-02-15 07:57:13,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:57:13,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:57:13,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:57:13,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23191.76 MB 2025-02-15 07:57:13,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25081.29 MB 2025-02-15 07:57:13,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:57:13,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 07:57:13,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 07:57:13,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:57:13,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26498.72 MB 2025-02-15 07:57:13,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:57:13,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:57:13,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:57:13,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25081.29 MB 2025-02-15 07:57:13,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27323.15 MB 2025-02-15 07:57:13,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 07:57:13,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 07:57:13,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34688.99 MB 2025-02-15 07:57:13,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:57:13,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32867.43 MB 2025-02-15 07:57:13,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:57:13,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:57:13,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:57:13,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23191.76 MB 2025-02-15 07:57:13,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27323.15 MB 2025-02-15 07:57:13,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 07:57:13,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 07:57:13,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34688.99 MB 2025-02-15 07:57:13,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 07:57:13,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32867.43 MB 2025-02-15 07:57:13,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:57:13,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:57:13,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:57:13,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28856.69 MB 2025-02-15 07:57:13,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29623.69 MB 2025-02-15 07:57:13,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:57:13,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34688.99 MB 2025-02-15 07:57:13,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35104.23 MB 2025-02-15 07:57:13,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 07:57:13,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30331.48 MB 2025-02-15 07:57:13,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:57:13,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:57:13,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:57:13,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30036.58 MB 2025-02-15 07:57:13,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30273.02 MB 2025-02-15 07:57:13,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.44 MB 2025-02-15 07:57:13,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35104.23 MB 2025-02-15 07:57:13,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35106.32 MB 2025-02-15 07:57:13,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 07:57:13,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30417.26 MB 2025-02-15 07:57:13,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:57:13,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:57:13,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.94 seconds 2025-02-15 07:57:13,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17581.63 MB 2025-02-15 07:57:13,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30474.10 MB 2025-02-15 07:57:13,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12892.47 MB 2025-02-15 07:57:13,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55855.55 MB 2025-02-15 07:57:13,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35106.32 MB 2025-02-15 07:57:13,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20749.22 MB 2025-02-15 07:57:13,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30474.10 MB 2025-02-15 07:57:13,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:57:13,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:57:13,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:57:13,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30474.10 MB 2025-02-15 07:57:13,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22586.02 MB 2025-02-15 07:57:13,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7888.08 MB 2025-02-15 07:57:13,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35106.32 MB 2025-02-15 07:57:13,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35106.32 MB 2025-02-15 07:57:13,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:57:13,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.76 MB 2025-02-15 07:57:13,974 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 07:57:13,974 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:57:13,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:57:13,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:57:13,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:57:13,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:13,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22586.02 MB 2025-02-15 07:57:13,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31025.04 MB 2025-02-15 07:57:13,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 07:57:13,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35106.32 MB 2025-02-15 07:57:13,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43497.03 MB 2025-02-15 07:57:13,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 07:57:13,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31025.04 MB 2025-02-15 07:57:14,140 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 07:57:14,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:14,142 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:57:14,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:14,143 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:57:14,147 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:57:14,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:14,148 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:57:14,149 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:57:31,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:31,267 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:57:31,272 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:57:31,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:31,276 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:57:31,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:31,277 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:57:49,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:57:49,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:57:49,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.66 seconds 2025-02-15 07:57:49,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:49,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-15 07:57:49,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-15 07:57:49,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-15 07:57:49,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56082.04 MB 2025-02-15 07:57:49,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29030.88 MB 2025-02-15 07:57:49,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27051.16 MB 2025-02-15 07:57:49,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34307.29 MB 2025-02-15 07:57:50,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:57:50,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:57:50,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 07:57:50,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:50,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-15 07:57:50,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21928.95 MB 2025-02-15 07:57:50,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3469.66 MB 2025-02-15 07:57:50,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29030.88 MB 2025-02-15 07:57:50,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39468.40 MB 2025-02-15 07:57:50,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10437.53 MB 2025-02-15 07:57:50,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37932.68 MB 2025-02-15 07:57:51,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:57:51,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:57:51,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 07:57:51,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:51,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.95 MB 2025-02-15 07:57:51,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22459.79 MB 2025-02-15 07:57:51,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 07:57:51,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39468.40 MB 2025-02-15 07:57:51,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26967.28 MB 2025-02-15 07:57:51,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12501.12 MB 2025-02-15 07:57:51,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26440.41 MB 2025-02-15 07:57:52,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:57:52,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:57:52,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:57:52,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-15 07:57:52,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24349.32 MB 2025-02-15 07:57:52,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 07:57:52,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26967.28 MB 2025-02-15 07:57:52,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27911.00 MB 2025-02-15 07:57:52,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 07:57:52,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25766.75 MB 2025-02-15 07:57:52,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:57:52,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:57:52,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 07:57:52,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24349.32 MB 2025-02-15 07:57:52,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26592.23 MB 2025-02-15 07:57:52,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 07:57:52,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27911.00 MB 2025-02-15 07:57:52,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 07:57:52,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 07:57:52,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.51 MB 2025-02-15 07:57:52,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:57:52,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:57:52,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 07:57:52,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-15 07:57:52,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26592.23 MB 2025-02-15 07:57:52,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 07:57:52,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26967.28 MB 2025-02-15 07:57:52,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 07:57:52,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 07:57:52,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.51 MB 2025-02-15 07:57:52,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:57:52,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:57:52,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:57:52,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28125.77 MB 2025-02-15 07:57:52,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.77 MB 2025-02-15 07:57:52,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 07:57:52,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-15 07:57:52,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34695.28 MB 2025-02-15 07:57:52,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 07:57:52,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29600.56 MB 2025-02-15 07:57:52,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:57:52,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:57:52,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:57:52,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29305.66 MB 2025-02-15 07:57:52,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29532.91 MB 2025-02-15 07:57:52,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.25 MB 2025-02-15 07:57:52,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34695.28 MB 2025-02-15 07:57:52,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34695.28 MB 2025-02-15 07:57:52,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:57:52,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29773.74 MB 2025-02-15 07:57:52,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:57:52,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:57:52,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.12 seconds 2025-02-15 07:57:52,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-15 07:57:52,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29733.77 MB 2025-02-15 07:57:52,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12643.39 MB 2025-02-15 07:57:52,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56082.04 MB 2025-02-15 07:57:52,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34695.28 MB 2025-02-15 07:57:52,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21386.76 MB 2025-02-15 07:57:52,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29773.74 MB 2025-02-15 07:57:52,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:57:52,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:57:52,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:57:52,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29733.77 MB 2025-02-15 07:57:52,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22081.36 MB 2025-02-15 07:57:52,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7652.41 MB 2025-02-15 07:57:52,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34695.28 MB 2025-02-15 07:57:52,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34695.28 MB 2025-02-15 07:57:52,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:57:52,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32234.07 MB 2025-02-15 07:57:52,684 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 07:57:52,684 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:57:52,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:57:52,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:57:52,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 07:57:52,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:57:52,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22081.36 MB 2025-02-15 07:57:52,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30482.30 MB 2025-02-15 07:57:52,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-15 07:57:52,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34695.28 MB 2025-02-15 07:57:52,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43048.24 MB 2025-02-15 07:57:52,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-15 07:57:52,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30482.30 MB 2025-02-15 07:57:52,847 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 07:57:52,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:52,848 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:57:52,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:52,849 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:57:52,854 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:57:52,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:57:52,855 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:57:52,855 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:58:47,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:47,411 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 07:58:47,416 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 07:58:47,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:47,420 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 331, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 07:58:47,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:47,421 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 331, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 07:58:52,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 07:58:52,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 07:58:52,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.17 seconds 2025-02-15 07:58:52,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15275.17 MB 2025-02-15 07:58:52,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16446.56 MB 2025-02-15 07:58:52,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.39 MB 2025-02-15 07:58:52,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55576.63 MB 2025-02-15 07:58:52,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 07:58:52,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36498.83 MB 2025-02-15 07:58:52,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25426.02 MB 2025-02-15 07:58:52,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 07:58:52,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 07:58:52,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:58:52,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16446.56 MB 2025-02-15 07:58:52,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.76 MB 2025-02-15 07:58:52,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1363.80 MB 2025-02-15 07:58:52,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 07:58:52,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 07:58:52,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:52,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17233.20 MB 2025-02-15 07:58:52,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 07:58:52,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 07:58:52,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 07:58:52,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.76 MB 2025-02-15 07:58:52,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15157.08 MB 2025-02-15 07:58:52,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 74.32 MB 2025-02-15 07:58:52,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 07:58:52,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19430.11 MB 2025-02-15 07:58:52,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 352.32 MB 2025-02-15 07:58:52,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18657.80 MB 2025-02-15 07:58:52,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 07:58:52,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 07:58:52,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:58:52,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-15 07:58:52,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15421.49 MB 2025-02-15 07:58:52,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.47 MB 2025-02-15 07:58:52,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19430.11 MB 2025-02-15 07:58:52,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19430.11 MB 2025-02-15 07:58:52,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:52,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15619.93 MB 2025-02-15 07:58:52,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 07:58:52,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 07:58:52,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 07:58:52,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.49 MB 2025-02-15 07:58:52,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15743.63 MB 2025-02-15 07:58:52,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.14 MB 2025-02-15 07:58:52,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19430.11 MB 2025-02-15 07:58:52,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19430.11 MB 2025-02-15 07:58:52,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:52,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16512.43 MB 2025-02-15 07:58:52,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 07:58:52,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 07:58:52,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 07:58:52,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-15 07:58:52,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15743.63 MB 2025-02-15 07:58:52,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 586.61 MB 2025-02-15 07:58:52,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19430.11 MB 2025-02-15 07:58:52,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19430.11 MB 2025-02-15 07:58:52,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:52,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16512.43 MB 2025-02-15 07:58:52,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 07:58:52,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 07:58:52,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 07:58:52,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16053.74 MB 2025-02-15 07:58:52,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16188.65 MB 2025-02-15 07:58:52,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.91 MB 2025-02-15 07:58:52,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19430.11 MB 2025-02-15 07:58:52,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 07:58:52,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 81.79 MB 2025-02-15 07:58:52,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16287.74 MB 2025-02-15 07:58:52,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 07:58:52,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 07:58:52,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 07:58:52,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16273.99 MB 2025-02-15 07:58:52,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16408.84 MB 2025-02-15 07:58:52,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.85 MB 2025-02-15 07:58:52,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 07:58:52,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 07:58:52,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:52,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16408.84 MB 2025-02-15 07:58:52,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 07:58:52,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 07:58:52,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.55 seconds 2025-02-15 07:58:52,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:52,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14121.94 MB 2025-02-15 07:58:52,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16530.35 MB 2025-02-15 07:58:52,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2408.42 MB 2025-02-15 07:58:52,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55576.63 MB 2025-02-15 07:58:52,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 07:58:52,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36064.72 MB 2025-02-15 07:58:52,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16530.35 MB 2025-02-15 07:58:53,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 07:58:53,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 07:58:53,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 07:58:53,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:53,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16530.35 MB 2025-02-15 07:58:53,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16269.97 MB 2025-02-15 07:58:53,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -260.39 MB 2025-02-15 07:58:53,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 07:58:53,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 07:58:53,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 07:58:53,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18412.52 MB 2025-02-15 07:58:53,147 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 4927, cut from 4929 2025-02-15 07:58:53,147 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 07:58:53,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 07:58:53,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 07:58:53,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 07:58:53,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 07:58:53,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16269.97 MB 2025-02-15 07:58:53,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21370.12 MB 2025-02-15 07:58:53,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5100.15 MB 2025-02-15 07:58:53,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 07:58:53,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25851.59 MB 2025-02-15 07:58:53,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6339.69 MB 2025-02-15 07:58:53,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21370.12 MB 2025-02-15 07:58:53,250 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 4719] 2025-02-15 07:58:53,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:53,251 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 07:58:53,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:53,252 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 07:58:53,257 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 07:58:53,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 07:58:53,258 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 07:58:53,258 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:00:44,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:00:44,601 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:00:44,606 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:00:44,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:00:44,610 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:00:44,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:00:44,611 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:01:02,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:01:02,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:01:02,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.05 seconds 2025-02-15 08:01:02,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:02,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21170.23 MB 2025-02-15 08:01:02,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25335.57 MB 2025-02-15 08:01:02,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4165.34 MB 2025-02-15 08:01:02,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30922.51 MB 2025-02-15 08:01:02,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28250.73 MB 2025-02-15 08:01:02,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2671.77 MB 2025-02-15 08:01:02,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34266.29 MB 2025-02-15 08:01:02,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:01:02,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:01:02,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:01:02,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:02,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25335.57 MB 2025-02-15 08:01:02,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21897.76 MB 2025-02-15 08:01:02,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3437.81 MB 2025-02-15 08:01:02,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28250.73 MB 2025-02-15 08:01:02,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38291.90 MB 2025-02-15 08:01:02,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10041.16 MB 2025-02-15 08:01:02,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37476.94 MB 2025-02-15 08:01:04,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:01:04,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:01:04,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-15 08:01:04,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:04,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21897.76 MB 2025-02-15 08:01:04,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22428.60 MB 2025-02-15 08:01:04,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:01:04,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38291.90 MB 2025-02-15 08:01:04,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24205.33 MB 2025-02-15 08:01:04,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14086.57 MB 2025-02-15 08:01:04,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26408.18 MB 2025-02-15 08:01:04,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:01:04,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:01:04,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:01:04,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:04,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22428.60 MB 2025-02-15 08:01:04,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24317.87 MB 2025-02-15 08:01:04,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-15 08:01:04,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24205.33 MB 2025-02-15 08:01:04,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27036.48 MB 2025-02-15 08:01:04,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:01:04,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25735.30 MB 2025-02-15 08:01:05,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:01:05,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:01:05,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:01:05,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24317.87 MB 2025-02-15 08:01:05,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26559.73 MB 2025-02-15 08:01:05,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:01:05,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27036.48 MB 2025-02-15 08:01:05,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33642.51 MB 2025-02-15 08:01:05,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:01:05,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32104.01 MB 2025-02-15 08:01:05,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:01:05,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:01:05,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:01:05,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22428.60 MB 2025-02-15 08:01:05,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26559.73 MB 2025-02-15 08:01:05,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-15 08:01:05,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24205.33 MB 2025-02-15 08:01:05,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33642.51 MB 2025-02-15 08:01:05,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-15 08:01:05,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32104.01 MB 2025-02-15 08:01:05,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:01:05,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:01:05,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:01:05,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28093.27 MB 2025-02-15 08:01:05,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28860.27 MB 2025-02-15 08:01:05,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:01:05,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33642.51 MB 2025-02-15 08:01:05,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 08:01:05,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:01:05,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29568.06 MB 2025-02-15 08:01:05,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:01:05,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:01:05,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:01:05,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29273.16 MB 2025-02-15 08:01:05,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29502.07 MB 2025-02-15 08:01:05,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 08:01:05,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34059.85 MB 2025-02-15 08:01:05,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 08:01:05,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:01:05,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29702.76 MB 2025-02-15 08:01:05,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:01:05,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:01:05,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.58 seconds 2025-02-15 08:01:05,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17069.47 MB 2025-02-15 08:01:05,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29703.09 MB 2025-02-15 08:01:05,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12633.63 MB 2025-02-15 08:01:05,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30922.51 MB 2025-02-15 08:01:05,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 08:01:05,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3137.34 MB 2025-02-15 08:01:05,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29703.09 MB 2025-02-15 08:01:05,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:01:05,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:01:05,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:01:05,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29703.09 MB 2025-02-15 08:01:05,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22073.10 MB 2025-02-15 08:01:05,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7630.00 MB 2025-02-15 08:01:05,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34059.85 MB 2025-02-15 08:01:05,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 08:01:05,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:01:05,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32214.15 MB 2025-02-15 08:01:05,482 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 08:01:05,482 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:01:05,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:01:05,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:01:05,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:01:05,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:01:05,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22073.10 MB 2025-02-15 08:01:05,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30510.57 MB 2025-02-15 08:01:05,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 08:01:05,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34059.85 MB 2025-02-15 08:01:05,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44545.61 MB 2025-02-15 08:01:05,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 08:01:05,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30510.57 MB 2025-02-15 08:01:05,659 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 08:01:05,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:01:05,660 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:01:05,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:01:05,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:01:05,666 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:01:05,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:01:05,667 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:01:05,667 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:02:01,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:01,529 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:02:01,535 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:02:01,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:01,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2470, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:02:01,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:01,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2470, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:02:40,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:02:40,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:02:40,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.60 seconds 2025-02-15 08:02:40,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:40,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30180.06 MB 2025-02-15 08:02:40,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38921.25 MB 2025-02-15 08:02:40,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8741.19 MB 2025-02-15 08:02:40,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70151.83 MB 2025-02-15 08:02:40,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42440.07 MB 2025-02-15 08:02:40,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27711.77 MB 2025-02-15 08:02:40,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47805.16 MB 2025-02-15 08:02:40,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:02:40,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:02:40,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-15 08:02:40,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:40,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38921.25 MB 2025-02-15 08:02:40,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28619.66 MB 2025-02-15 08:02:40,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10301.59 MB 2025-02-15 08:02:40,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42440.07 MB 2025-02-15 08:02:40,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61509.47 MB 2025-02-15 08:02:40,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19069.40 MB 2025-02-15 08:02:40,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64397.55 MB 2025-02-15 08:02:42,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:02:42,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:02:42,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:02:42,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28619.66 MB 2025-02-15 08:02:42,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29150.50 MB 2025-02-15 08:02:42,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:02:42,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61509.47 MB 2025-02-15 08:02:42,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31448.89 MB 2025-02-15 08:02:42,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30060.58 MB 2025-02-15 08:02:42,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33129.05 MB 2025-02-15 08:02:42,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:02:42,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:02:42,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:02:42,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29150.50 MB 2025-02-15 08:02:42,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31039.90 MB 2025-02-15 08:02:42,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.40 MB 2025-02-15 08:02:42,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31448.89 MB 2025-02-15 08:02:42,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34280.05 MB 2025-02-15 08:02:42,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:02:42,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32457.33 MB 2025-02-15 08:02:42,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:02:42,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:02:42,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:02:42,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31039.90 MB 2025-02-15 08:02:42,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33281.76 MB 2025-02-15 08:02:42,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:02:42,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34280.05 MB 2025-02-15 08:02:42,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40414.22 MB 2025-02-15 08:02:42,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:02:42,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38826.04 MB 2025-02-15 08:02:42,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:02:42,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:02:42,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:02:42,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29150.50 MB 2025-02-15 08:02:42,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33281.76 MB 2025-02-15 08:02:42,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.26 MB 2025-02-15 08:02:42,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31448.89 MB 2025-02-15 08:02:42,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40414.22 MB 2025-02-15 08:02:42,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:02:42,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38826.04 MB 2025-02-15 08:02:42,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:02:42,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:02:42,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:02:42,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34815.30 MB 2025-02-15 08:02:42,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35582.30 MB 2025-02-15 08:02:42,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:02:42,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40414.22 MB 2025-02-15 08:02:42,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40829.45 MB 2025-02-15 08:02:42,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:02:42,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36290.09 MB 2025-02-15 08:02:42,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:02:42,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:02:42,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:02:42,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35995.19 MB 2025-02-15 08:02:42,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36223.22 MB 2025-02-15 08:02:42,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-15 08:02:42,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40829.45 MB 2025-02-15 08:02:42,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40829.45 MB 2025-02-15 08:02:42,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:02:42,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36422.99 MB 2025-02-15 08:02:42,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:02:42,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:02:42,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.31 seconds 2025-02-15 08:02:42,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:42,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21574.38 MB 2025-02-15 08:02:42,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36423.16 MB 2025-02-15 08:02:42,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14848.78 MB 2025-02-15 08:02:42,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61543.02 MB 2025-02-15 08:02:42,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40829.45 MB 2025-02-15 08:02:42,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20713.57 MB 2025-02-15 08:02:42,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36423.16 MB 2025-02-15 08:02:43,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:02:43,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:02:43,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:02:43,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:43,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36423.16 MB 2025-02-15 08:02:43,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26562.16 MB 2025-02-15 08:02:43,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9861.00 MB 2025-02-15 08:02:43,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40829.45 MB 2025-02-15 08:02:43,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40829.45 MB 2025-02-15 08:02:43,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:02:43,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38920.70 MB 2025-02-15 08:02:43,139 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 08:02:43,139 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:02:43,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:02:43,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:02:43,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:02:43,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:02:43,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26562.16 MB 2025-02-15 08:02:43,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34953.72 MB 2025-02-15 08:02:43,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.56 MB 2025-02-15 08:02:43,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40829.45 MB 2025-02-15 08:02:43,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45000.69 MB 2025-02-15 08:02:43,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 08:02:43,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34953.72 MB 2025-02-15 08:02:43,302 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 08:02:43,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:43,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:02:43,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:43,304 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:02:43,309 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:02:43,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:02:43,310 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:02:43,310 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:03:51,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:03:51,880 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:03:51,885 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:03:51,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:03:51,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:03:51,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:03:51,890 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:04:13,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:04:13,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:04:13,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.21 seconds 2025-02-15 08:04:13,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:13,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-15 08:04:13,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-15 08:04:13,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-15 08:04:13,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53343.16 MB 2025-02-15 08:04:13,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38080.09 MB 2025-02-15 08:04:13,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15263.07 MB 2025-02-15 08:04:13,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-15 08:04:13,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:04:13,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:04:13,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 08:04:13,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:13,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-15 08:04:13,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-15 08:04:13,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-15 08:04:13,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38080.09 MB 2025-02-15 08:04:13,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47611.64 MB 2025-02-15 08:04:13,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9531.56 MB 2025-02-15 08:04:13,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41696.53 MB 2025-02-15 08:04:15,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:04:15,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:04:15,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:04:15,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-15 08:04:15,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-15 08:04:15,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:04:15,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47611.64 MB 2025-02-15 08:04:15,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 08:04:15,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18551.41 MB 2025-02-15 08:04:15,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27409.44 MB 2025-02-15 08:04:15,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:04:15,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:04:15,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:04:15,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-15 08:04:15,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-15 08:04:15,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:04:15,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 08:04:15,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30003.95 MB 2025-02-15 08:04:15,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 08:04:15,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-15 08:04:15,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:04:15,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:04:15,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 08:04:15,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-15 08:04:15,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-15 08:04:15,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:04:15,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30003.95 MB 2025-02-15 08:04:15,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35666.26 MB 2025-02-15 08:04:15,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:04:15,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-15 08:04:15,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:04:15,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:04:15,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:04:15,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-15 08:04:15,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-15 08:04:15,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:04:15,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 08:04:15,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35666.26 MB 2025-02-15 08:04:15,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:04:15,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-15 08:04:15,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:04:15,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:04:15,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:04:15,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-15 08:04:15,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29862.83 MB 2025-02-15 08:04:15,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:04:15,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35666.26 MB 2025-02-15 08:04:15,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36079.40 MB 2025-02-15 08:04:15,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 08:04:15,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.62 MB 2025-02-15 08:04:15,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:04:15,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:04:15,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:04:15,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30275.72 MB 2025-02-15 08:04:15,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30504.51 MB 2025-02-15 08:04:15,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 08:04:15,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36079.40 MB 2025-02-15 08:04:15,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36079.40 MB 2025-02-15 08:04:15,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:04:15,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30741.66 MB 2025-02-15 08:04:15,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:04:15,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:04:15,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.68 seconds 2025-02-15 08:04:15,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-15 08:04:15,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30705.36 MB 2025-02-15 08:04:15,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12963.46 MB 2025-02-15 08:04:15,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53343.16 MB 2025-02-15 08:04:15,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36079.40 MB 2025-02-15 08:04:15,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17263.76 MB 2025-02-15 08:04:15,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30741.66 MB 2025-02-15 08:04:15,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:04:15,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:04:15,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:04:15,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30705.36 MB 2025-02-15 08:04:15,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22741.07 MB 2025-02-15 08:04:15,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7964.28 MB 2025-02-15 08:04:15,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36079.40 MB 2025-02-15 08:04:15,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36079.40 MB 2025-02-15 08:04:15,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:04:15,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33212.73 MB 2025-02-15 08:04:15,855 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 08:04:15,855 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:04:15,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:04:15,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:04:15,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:04:15,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:04:15,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22741.07 MB 2025-02-15 08:04:15,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31166.03 MB 2025-02-15 08:04:15,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 08:04:15,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36079.40 MB 2025-02-15 08:04:15,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44455.43 MB 2025-02-15 08:04:15,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 08:04:15,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31166.03 MB 2025-02-15 08:04:16,020 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 08:04:16,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:04:16,022 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:04:16,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:04:16,022 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:04:16,027 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:04:16,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:04:16,028 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:04:16,028 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:05:31,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:05:31,770 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:05:31,775 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:05:31,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:05:31,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1708, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:05:31,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:05:31,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1708, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:05:58,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:05:58,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:05:58,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.46 seconds 2025-02-15 08:05:58,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:05:58,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24870.32 MB 2025-02-15 08:05:58,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30914.84 MB 2025-02-15 08:05:58,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6044.52 MB 2025-02-15 08:05:58,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52831.45 MB 2025-02-15 08:05:58,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39244.01 MB 2025-02-15 08:05:58,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13587.45 MB 2025-02-15 08:05:58,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39777.51 MB 2025-02-15 08:05:58,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:05:58,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:05:58,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 08:05:58,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:05:58,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30914.84 MB 2025-02-15 08:05:58,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24657.21 MB 2025-02-15 08:05:58,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6257.63 MB 2025-02-15 08:05:58,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39244.01 MB 2025-02-15 08:05:58,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47735.37 MB 2025-02-15 08:05:58,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8491.37 MB 2025-02-15 08:05:58,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42212.53 MB 2025-02-15 08:06:00,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:06:00,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:06:00,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:06:00,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24657.21 MB 2025-02-15 08:06:00,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25188.05 MB 2025-02-15 08:06:00,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:06:00,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47735.37 MB 2025-02-15 08:06:00,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30442.26 MB 2025-02-15 08:06:00,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17293.12 MB 2025-02-15 08:06:00,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29166.60 MB 2025-02-15 08:06:00,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:06:00,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:06:00,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:06:00,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25188.05 MB 2025-02-15 08:06:00,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27077.59 MB 2025-02-15 08:06:00,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:06:00,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 08:06:00,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30442.26 MB 2025-02-15 08:06:00,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:06:00,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28495.01 MB 2025-02-15 08:06:00,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:06:00,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:06:00,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:06:00,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27077.59 MB 2025-02-15 08:06:00,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29319.44 MB 2025-02-15 08:06:00,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:06:00,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 08:06:00,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37048.29 MB 2025-02-15 08:06:00,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:06:00,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34863.72 MB 2025-02-15 08:06:00,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:06:00,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:06:00,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:06:00,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25188.05 MB 2025-02-15 08:06:00,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29319.44 MB 2025-02-15 08:06:00,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:06:00,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 08:06:00,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37048.29 MB 2025-02-15 08:06:00,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:06:00,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34863.72 MB 2025-02-15 08:06:00,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:06:00,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:06:00,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:06:00,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30852.98 MB 2025-02-15 08:06:00,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31619.99 MB 2025-02-15 08:06:00,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:06:00,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37048.29 MB 2025-02-15 08:06:00,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 08:06:00,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:06:00,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32327.77 MB 2025-02-15 08:06:00,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:06:00,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:06:00,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:06:00,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32032.88 MB 2025-02-15 08:06:00,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32259.65 MB 2025-02-15 08:06:00,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.77 MB 2025-02-15 08:06:00,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 08:06:00,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 08:06:00,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:06:00,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32493.59 MB 2025-02-15 08:06:00,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:06:00,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:06:00,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.89 seconds 2025-02-15 08:06:00,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18919.52 MB 2025-02-15 08:06:00,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32460.50 MB 2025-02-15 08:06:00,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13540.98 MB 2025-02-15 08:06:00,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52831.45 MB 2025-02-15 08:06:00,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 08:06:00,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15367.93 MB 2025-02-15 08:06:00,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32493.59 MB 2025-02-15 08:06:00,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:06:00,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:06:00,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:06:00,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32460.50 MB 2025-02-15 08:06:00,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23914.42 MB 2025-02-15 08:06:00,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8546.08 MB 2025-02-15 08:06:00,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 08:06:00,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37463.52 MB 2025-02-15 08:06:00,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:06:00,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34964.18 MB 2025-02-15 08:06:00,964 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 08:06:00,965 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:06:00,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:06:00,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:06:00,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:06:00,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:06:00,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23914.42 MB 2025-02-15 08:06:00,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32326.85 MB 2025-02-15 08:06:00,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-15 08:06:00,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37463.52 MB 2025-02-15 08:06:00,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45826.97 MB 2025-02-15 08:06:00,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 08:06:00,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32326.85 MB 2025-02-15 08:06:01,136 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 08:06:01,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:06:01,137 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:06:01,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:06:01,138 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:06:01,143 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:06:01,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:06:01,144 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:06:01,144 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:08:01,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:01,306 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:08:01,315 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:08:01,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:01,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1825, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:08:01,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:01,326 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1825, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:08:29,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:08:29,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:08:29,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.14 seconds 2025-02-15 08:08:29,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:29,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25685.60 MB 2025-02-15 08:08:29,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32144.83 MB 2025-02-15 08:08:29,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6459.23 MB 2025-02-15 08:08:29,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54190.41 MB 2025-02-15 08:08:29,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39655.05 MB 2025-02-15 08:08:29,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14535.36 MB 2025-02-15 08:08:29,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41045.77 MB 2025-02-15 08:08:29,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:08:29,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:08:29,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 08:08:29,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:29,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32144.83 MB 2025-02-15 08:08:29,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25265.46 MB 2025-02-15 08:08:29,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6879.37 MB 2025-02-15 08:08:29,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39655.05 MB 2025-02-15 08:08:29,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53890.51 MB 2025-02-15 08:08:29,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14235.47 MB 2025-02-15 08:08:29,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51422.21 MB 2025-02-15 08:08:31,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:08:31,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:08:31,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:08:31,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25265.46 MB 2025-02-15 08:08:31,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25796.30 MB 2025-02-15 08:08:31,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:08:31,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 08:08:31,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34611.40 MB 2025-02-15 08:08:31,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19279.12 MB 2025-02-15 08:08:31,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29774.84 MB 2025-02-15 08:08:31,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:08:31,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:08:31,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:08:31,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25796.30 MB 2025-02-15 08:08:31,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27685.83 MB 2025-02-15 08:08:31,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:08:31,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34611.40 MB 2025-02-15 08:08:31,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34611.40 MB 2025-02-15 08:08:31,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:08:31,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29103.26 MB 2025-02-15 08:08:31,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:08:31,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:08:31,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:08:31,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27685.83 MB 2025-02-15 08:08:31,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.69 MB 2025-02-15 08:08:31,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:08:31,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34611.40 MB 2025-02-15 08:08:31,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38386.27 MB 2025-02-15 08:08:31,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:08:31,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.97 MB 2025-02-15 08:08:31,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:08:31,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:08:31,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:08:31,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25796.30 MB 2025-02-15 08:08:31,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.69 MB 2025-02-15 08:08:31,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:08:31,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34611.40 MB 2025-02-15 08:08:31,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38386.27 MB 2025-02-15 08:08:31,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:08:31,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.97 MB 2025-02-15 08:08:31,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:08:31,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:08:31,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:08:31,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31461.23 MB 2025-02-15 08:08:31,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32228.23 MB 2025-02-15 08:08:31,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:08:31,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38386.27 MB 2025-02-15 08:08:31,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38801.51 MB 2025-02-15 08:08:31,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:08:31,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32936.02 MB 2025-02-15 08:08:31,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:08:31,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:08:31,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:08:31,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32641.12 MB 2025-02-15 08:08:31,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32870.27 MB 2025-02-15 08:08:31,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.15 MB 2025-02-15 08:08:31,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38801.51 MB 2025-02-15 08:08:31,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38801.51 MB 2025-02-15 08:08:31,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:08:31,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33112.39 MB 2025-02-15 08:08:31,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:08:31,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:08:31,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.64 seconds 2025-02-15 08:08:31,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:31,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19327.15 MB 2025-02-15 08:08:31,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33071.13 MB 2025-02-15 08:08:31,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13743.97 MB 2025-02-15 08:08:31,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54190.41 MB 2025-02-15 08:08:31,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38801.51 MB 2025-02-15 08:08:31,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15388.90 MB 2025-02-15 08:08:31,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33112.39 MB 2025-02-15 08:08:32,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:08:32,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:08:32,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:08:32,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:32,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33071.13 MB 2025-02-15 08:08:32,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24319.92 MB 2025-02-15 08:08:32,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8751.21 MB 2025-02-15 08:08:32,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38801.51 MB 2025-02-15 08:08:32,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38801.51 MB 2025-02-15 08:08:32,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:08:32,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35572.96 MB 2025-02-15 08:08:32,258 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 08:08:32,259 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:08:32,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:08:32,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:08:32,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:08:32,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:08:32,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24319.92 MB 2025-02-15 08:08:32,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32725.58 MB 2025-02-15 08:08:32,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 08:08:32,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38801.51 MB 2025-02-15 08:08:32,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42981.13 MB 2025-02-15 08:08:32,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 08:08:32,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32725.58 MB 2025-02-15 08:08:32,422 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 08:08:32,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:32,423 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:08:32,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:32,424 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:08:32,429 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:08:32,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:32,430 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:08:32,430 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:08:53,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:53,192 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:08:53,197 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:08:53,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:53,200 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2463, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:08:53,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:08:53,201 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2463, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:09:31,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:09:31,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:09:31,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.38 seconds 2025-02-15 08:09:31,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:31,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30132.45 MB 2025-02-15 08:09:31,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38848.87 MB 2025-02-15 08:09:31,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8716.42 MB 2025-02-15 08:09:31,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68507.66 MB 2025-02-15 08:09:31,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42364.57 MB 2025-02-15 08:09:31,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26143.10 MB 2025-02-15 08:09:31,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47757.55 MB 2025-02-15 08:09:31,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:09:31,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:09:31,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 08:09:31,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:31,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38848.87 MB 2025-02-15 08:09:31,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28584.44 MB 2025-02-15 08:09:31,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10264.44 MB 2025-02-15 08:09:31,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42364.57 MB 2025-02-15 08:09:31,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61459.14 MB 2025-02-15 08:09:31,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19094.57 MB 2025-02-15 08:09:31,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64369.34 MB 2025-02-15 08:09:33,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:09:33,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:09:33,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 08:09:33,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:33,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28584.44 MB 2025-02-15 08:09:33,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29115.28 MB 2025-02-15 08:09:33,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:09:33,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61459.14 MB 2025-02-15 08:09:33,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31411.14 MB 2025-02-15 08:09:33,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30047.99 MB 2025-02-15 08:09:33,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33093.83 MB 2025-02-15 08:09:33,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:09:33,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:09:33,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:09:33,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:33,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29115.28 MB 2025-02-15 08:09:33,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31004.48 MB 2025-02-15 08:09:33,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-15 08:09:33,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31411.14 MB 2025-02-15 08:09:33,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-15 08:09:33,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:09:33,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32421.91 MB 2025-02-15 08:09:34,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:09:34,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:09:34,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 08:09:34,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31004.48 MB 2025-02-15 08:09:34,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33246.34 MB 2025-02-15 08:09:34,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:09:34,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34242.30 MB 2025-02-15 08:09:34,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40376.47 MB 2025-02-15 08:09:34,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:09:34,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38790.62 MB 2025-02-15 08:09:34,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:09:34,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:09:34,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 08:09:34,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29115.28 MB 2025-02-15 08:09:34,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33246.34 MB 2025-02-15 08:09:34,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-15 08:09:34,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31411.14 MB 2025-02-15 08:09:34,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40376.47 MB 2025-02-15 08:09:34,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:09:34,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38790.62 MB 2025-02-15 08:09:34,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:09:34,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:09:34,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:09:34,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34779.88 MB 2025-02-15 08:09:34,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35546.88 MB 2025-02-15 08:09:34,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:09:34,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40376.47 MB 2025-02-15 08:09:34,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40791.70 MB 2025-02-15 08:09:34,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:09:34,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36254.67 MB 2025-02-15 08:09:34,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:09:34,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:09:34,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:09:34,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35959.77 MB 2025-02-15 08:09:34,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36188.87 MB 2025-02-15 08:09:34,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-15 08:09:34,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40791.70 MB 2025-02-15 08:09:34,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40791.70 MB 2025-02-15 08:09:34,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:09:34,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36413.32 MB 2025-02-15 08:09:34,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:09:34,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:09:34,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.21 seconds 2025-02-15 08:09:34,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21550.58 MB 2025-02-15 08:09:34,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36389.45 MB 2025-02-15 08:09:34,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14838.87 MB 2025-02-15 08:09:34,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59924.02 MB 2025-02-15 08:09:34,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40791.70 MB 2025-02-15 08:09:34,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19132.32 MB 2025-02-15 08:09:34,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36413.32 MB 2025-02-15 08:09:34,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:09:34,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:09:34,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 08:09:34,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36389.45 MB 2025-02-15 08:09:34,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26547.62 MB 2025-02-15 08:09:34,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9841.83 MB 2025-02-15 08:09:34,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40791.70 MB 2025-02-15 08:09:34,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40791.70 MB 2025-02-15 08:09:34,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:09:34,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38894.97 MB 2025-02-15 08:09:34,709 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 08:09:34,710 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:09:34,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:09:34,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:09:34,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:09:34,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:09:34,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26547.62 MB 2025-02-15 08:09:34,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34965.67 MB 2025-02-15 08:09:34,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.05 MB 2025-02-15 08:09:34,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40791.70 MB 2025-02-15 08:09:34,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44977.62 MB 2025-02-15 08:09:34,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 08:09:34,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34965.67 MB 2025-02-15 08:09:34,872 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 08:09:34,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:09:34,874 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:09:34,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:09:34,875 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:09:34,879 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:09:34,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:09:34,880 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:09:34,881 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:10:49,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:49,294 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:10:49,299 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:10:49,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:49,303 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:10:49,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:49,304 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:10:55,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:10:55,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:10:55,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.99 seconds 2025-02-15 08:10:55,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:55,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15658.42 MB 2025-02-15 08:10:55,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17024.45 MB 2025-02-15 08:10:55,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1366.03 MB 2025-02-15 08:10:55,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53345.26 MB 2025-02-15 08:10:55,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20715.67 MB 2025-02-15 08:10:55,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32629.59 MB 2025-02-15 08:10:55,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26036.28 MB 2025-02-15 08:10:55,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:10:55,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:10:55,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:10:55,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:55,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17024.45 MB 2025-02-15 08:10:55,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17055.20 MB 2025-02-15 08:10:55,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 30.75 MB 2025-02-15 08:10:55,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20715.67 MB 2025-02-15 08:10:55,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23322.43 MB 2025-02-15 08:10:55,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2606.76 MB 2025-02-15 08:10:55,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21184.17 MB 2025-02-15 08:10:56,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:10:56,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:10:56,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.42 seconds 2025-02-15 08:10:56,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:56,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17055.20 MB 2025-02-15 08:10:56,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17448.02 MB 2025-02-15 08:10:56,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.82 MB 2025-02-15 08:10:56,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23322.43 MB 2025-02-15 08:10:56,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19692.26 MB 2025-02-15 08:10:56,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3630.17 MB 2025-02-15 08:10:56,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21395.76 MB 2025-02-15 08:10:56,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:10:56,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:10:56,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:10:56,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:56,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17448.02 MB 2025-02-15 08:10:56,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18846.81 MB 2025-02-15 08:10:56,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1398.79 MB 2025-02-15 08:10:56,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19692.26 MB 2025-02-15 08:10:56,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21787.31 MB 2025-02-15 08:10:56,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2095.05 MB 2025-02-15 08:10:56,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19895.71 MB 2025-02-15 08:10:56,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:10:56,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:10:56,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:10:56,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:56,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18846.81 MB 2025-02-15 08:10:56,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20505.80 MB 2025-02-15 08:10:56,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1658.99 MB 2025-02-15 08:10:56,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21787.31 MB 2025-02-15 08:10:56,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25979.52 MB 2025-02-15 08:10:56,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 08:10:56,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24610.65 MB 2025-02-15 08:10:56,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:10:56,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:10:56,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:10:56,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:56,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17448.02 MB 2025-02-15 08:10:56,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20505.80 MB 2025-02-15 08:10:56,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3057.78 MB 2025-02-15 08:10:56,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19692.26 MB 2025-02-15 08:10:56,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25979.52 MB 2025-02-15 08:10:56,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6287.26 MB 2025-02-15 08:10:56,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24610.65 MB 2025-02-15 08:10:57,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:10:57,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:10:57,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 08:10:57,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:57,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21641.67 MB 2025-02-15 08:10:57,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22210.30 MB 2025-02-15 08:10:57,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 568.63 MB 2025-02-15 08:10:57,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25979.52 MB 2025-02-15 08:10:57,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 08:10:57,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 304.09 MB 2025-02-15 08:10:57,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22734.06 MB 2025-02-15 08:10:57,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:10:57,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:10:57,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:10:57,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:57,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.84 MB 2025-02-15 08:10:57,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22743.98 MB 2025-02-15 08:10:57,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.14 MB 2025-02-15 08:10:57,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 08:10:57,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 08:10:57,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:10:57,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22836.25 MB 2025-02-15 08:10:57,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:10:57,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:10:57,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.76 seconds 2025-02-15 08:10:57,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:57,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-15 08:10:57,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22945.05 MB 2025-02-15 08:10:57,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8631.49 MB 2025-02-15 08:10:57,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53345.26 MB 2025-02-15 08:10:57,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 08:10:57,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27061.65 MB 2025-02-15 08:10:57,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22945.05 MB 2025-02-15 08:10:57,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:10:57,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:10:57,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:10:57,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:57,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22945.05 MB 2025-02-15 08:10:57,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25959.08 MB 2025-02-15 08:10:57,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 08:10:57,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 08:10:57,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27625.78 MB 2025-02-15 08:10:57,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-15 08:10:57,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26260.71 MB 2025-02-15 08:10:57,352 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:10:57,352 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 08:10:57,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:10:57,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:10:57,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:10:57,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:10:57,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18829.24 MB 2025-02-15 08:10:57,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27268.26 MB 2025-02-15 08:10:57,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:10:57,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27625.78 MB 2025-02-15 08:10:57,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38115.74 MB 2025-02-15 08:10:57,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:10:57,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27268.26 MB 2025-02-15 08:10:57,519 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:10:57,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:57,520 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:10:57,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:57,521 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:10:57,526 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:10:57,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:10:57,527 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:10:57,527 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 08:11:35,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:11:35,042 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:11:35,047 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:11:35,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:11:35,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1716, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:11:35,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:11:35,053 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1716, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:12:01,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:12:01,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:12:01,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.58 seconds 2025-02-15 08:12:01,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:01,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24926.07 MB 2025-02-15 08:12:01,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30999.42 MB 2025-02-15 08:12:01,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6073.35 MB 2025-02-15 08:12:01,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50700.75 MB 2025-02-15 08:12:01,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-15 08:12:01,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11368.66 MB 2025-02-15 08:12:01,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39833.26 MB 2025-02-15 08:12:01,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:12:01,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:12:01,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 08:12:01,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:01,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30999.42 MB 2025-02-15 08:12:01,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24698.80 MB 2025-02-15 08:12:01,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6300.62 MB 2025-02-15 08:12:01,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39332.09 MB 2025-02-15 08:12:01,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52282.00 MB 2025-02-15 08:12:01,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12949.91 MB 2025-02-15 08:12:01,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48343.51 MB 2025-02-15 08:12:03,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:12:03,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:12:03,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:12:03,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:03,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24698.80 MB 2025-02-15 08:12:03,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25229.64 MB 2025-02-15 08:12:03,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:12:03,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52282.00 MB 2025-02-15 08:12:03,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 08:12:03,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17607.69 MB 2025-02-15 08:12:03,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29208.19 MB 2025-02-15 08:12:03,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:12:03,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:12:03,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:12:03,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:03,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25229.64 MB 2025-02-15 08:12:03,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27119.17 MB 2025-02-15 08:12:03,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:12:03,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 08:12:03,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 08:12:03,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:03,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28536.60 MB 2025-02-15 08:12:03,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:12:03,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:12:03,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:12:03,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:03,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27119.17 MB 2025-02-15 08:12:03,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.03 MB 2025-02-15 08:12:03,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:12:03,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 08:12:03,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37505.47 MB 2025-02-15 08:12:03,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:12:03,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34905.31 MB 2025-02-15 08:12:03,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:12:03,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:12:03,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:12:03,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:03,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25229.64 MB 2025-02-15 08:12:03,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.03 MB 2025-02-15 08:12:03,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:12:03,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 08:12:03,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37505.47 MB 2025-02-15 08:12:03,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:12:03,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34905.31 MB 2025-02-15 08:12:04,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:12:04,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:12:04,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 08:12:04,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:04,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30894.57 MB 2025-02-15 08:12:04,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31661.57 MB 2025-02-15 08:12:04,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:12:04,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37505.47 MB 2025-02-15 08:12:04,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 08:12:04,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:12:04,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32369.36 MB 2025-02-15 08:12:04,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:12:04,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:12:04,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:12:04,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:04,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32074.46 MB 2025-02-15 08:12:04,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32300.93 MB 2025-02-15 08:12:04,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.47 MB 2025-02-15 08:12:04,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 08:12:04,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 08:12:04,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:04,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.30 MB 2025-02-15 08:12:04,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:12:04,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:12:04,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.08 seconds 2025-02-15 08:12:04,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:04,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18947.39 MB 2025-02-15 08:12:04,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32501.78 MB 2025-02-15 08:12:04,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13554.39 MB 2025-02-15 08:12:04,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50700.75 MB 2025-02-15 08:12:04,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 08:12:04,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12780.04 MB 2025-02-15 08:12:04,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.30 MB 2025-02-15 08:12:04,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:12:04,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:12:04,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:12:04,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:04,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32501.78 MB 2025-02-15 08:12:04,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23938.01 MB 2025-02-15 08:12:04,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8563.77 MB 2025-02-15 08:12:04,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 08:12:04,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 08:12:04,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:04,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35001.78 MB 2025-02-15 08:12:04,423 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 08:12:04,423 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:12:04,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:12:04,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:12:04,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:12:04,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:04,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23938.01 MB 2025-02-15 08:12:04,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32338.43 MB 2025-02-15 08:12:04,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.42 MB 2025-02-15 08:12:04,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 08:12:04,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42096.13 MB 2025-02-15 08:12:04,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 08:12:04,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32338.43 MB 2025-02-15 08:12:04,589 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 08:12:04,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:04,591 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:12:04,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:04,592 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:12:04,596 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:12:04,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:04,598 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:12:04,598 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:12:30,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:30,273 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:12:30,278 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:12:30,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:30,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 852, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:12:30,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:30,283 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 852, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:12:43,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:12:43,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:12:43,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.26 seconds 2025-02-15 08:12:43,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:43,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18905.58 MB 2025-02-15 08:12:43,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21921.28 MB 2025-02-15 08:12:43,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3015.70 MB 2025-02-15 08:12:43,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50446.99 MB 2025-02-15 08:12:43,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27845.98 MB 2025-02-15 08:12:43,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22601.01 MB 2025-02-15 08:12:43,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30868.37 MB 2025-02-15 08:12:43,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:12:43,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:12:43,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 08:12:43,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:43,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21921.28 MB 2025-02-15 08:12:43,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20207.13 MB 2025-02-15 08:12:43,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1714.15 MB 2025-02-15 08:12:43,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27845.98 MB 2025-02-15 08:12:43,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35152.46 MB 2025-02-15 08:12:43,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7306.48 MB 2025-02-15 08:12:43,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32031.36 MB 2025-02-15 08:12:45,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:12:45,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:12:45,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:12:45,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20207.13 MB 2025-02-15 08:12:45,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20737.97 MB 2025-02-15 08:12:45,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:12:45,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35152.46 MB 2025-02-15 08:12:45,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26245.86 MB 2025-02-15 08:12:45,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8906.60 MB 2025-02-15 08:12:45,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24716.52 MB 2025-02-15 08:12:45,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:12:45,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:12:45,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:12:45,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-15 08:12:45,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22627.51 MB 2025-02-15 08:12:45,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:12:45,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26245.86 MB 2025-02-15 08:12:45,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26245.86 MB 2025-02-15 08:12:45,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:45,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24044.94 MB 2025-02-15 08:12:45,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:12:45,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:12:45,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:12:45,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22627.51 MB 2025-02-15 08:12:45,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-15 08:12:45,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:12:45,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26245.86 MB 2025-02-15 08:12:45,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-15 08:12:45,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:12:45,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-15 08:12:45,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:12:45,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:12:45,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:12:45,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-15 08:12:45,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-15 08:12:45,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:12:45,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26245.86 MB 2025-02-15 08:12:45,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-15 08:12:45,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:12:45,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-15 08:12:45,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:12:45,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:12:45,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:12:45,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26402.91 MB 2025-02-15 08:12:45,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27169.91 MB 2025-02-15 08:12:45,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:12:45,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32380.03 MB 2025-02-15 08:12:45,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32795.26 MB 2025-02-15 08:12:45,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:12:45,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.70 MB 2025-02-15 08:12:45,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:12:45,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:12:45,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:12:45,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27582.80 MB 2025-02-15 08:12:45,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27810.46 MB 2025-02-15 08:12:45,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.66 MB 2025-02-15 08:12:45,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32795.26 MB 2025-02-15 08:12:45,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32795.26 MB 2025-02-15 08:12:45,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:45,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28030.63 MB 2025-02-15 08:12:45,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:12:45,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:12:45,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.65 seconds 2025-02-15 08:12:45,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:45,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15937.14 MB 2025-02-15 08:12:45,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28011.33 MB 2025-02-15 08:12:45,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12074.19 MB 2025-02-15 08:12:45,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50446.99 MB 2025-02-15 08:12:45,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32795.26 MB 2025-02-15 08:12:45,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17651.73 MB 2025-02-15 08:12:45,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28030.63 MB 2025-02-15 08:12:46,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:12:46,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:12:46,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:12:46,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:46,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28011.33 MB 2025-02-15 08:12:46,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20938.48 MB 2025-02-15 08:12:46,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7072.85 MB 2025-02-15 08:12:46,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32795.26 MB 2025-02-15 08:12:46,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32795.26 MB 2025-02-15 08:12:46,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:12:46,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.54 MB 2025-02-15 08:12:46,231 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 08:12:46,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:12:46,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:12:46,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:12:46,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:12:46,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:12:46,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20938.48 MB 2025-02-15 08:12:46,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29369.16 MB 2025-02-15 08:12:46,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 08:12:46,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32795.26 MB 2025-02-15 08:12:46,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41177.58 MB 2025-02-15 08:12:46,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 08:12:46,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29369.16 MB 2025-02-15 08:12:46,399 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 08:12:46,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:46,401 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:12:46,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:46,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:12:46,406 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:12:46,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:12:46,407 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:12:46,407 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:13:58,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:13:58,613 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:13:58,618 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:13:58,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:13:58,622 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 420, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:13:58,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:13:58,623 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 420, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:14:05,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:14:05,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:14:05,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.47 seconds 2025-02-15 08:14:05,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:05,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.33 MB 2025-02-15 08:14:05,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17381.69 MB 2025-02-15 08:14:05,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1486.36 MB 2025-02-15 08:14:05,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53750.01 MB 2025-02-15 08:14:05,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20254.29 MB 2025-02-15 08:14:05,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33495.71 MB 2025-02-15 08:14:05,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26272.67 MB 2025-02-15 08:14:05,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:14:05,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:14:05,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:14:05,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:05,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17381.69 MB 2025-02-15 08:14:05,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17962.35 MB 2025-02-15 08:14:05,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 580.66 MB 2025-02-15 08:14:05,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20254.29 MB 2025-02-15 08:14:05,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25226.64 MB 2025-02-15 08:14:05,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4972.35 MB 2025-02-15 08:14:05,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24452.44 MB 2025-02-15 08:14:07,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:14:07,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:14:07,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:14:07,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17962.35 MB 2025-02-15 08:14:07,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18493.19 MB 2025-02-15 08:14:07,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:14:07,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25226.64 MB 2025-02-15 08:14:07,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21434.99 MB 2025-02-15 08:14:07,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3791.65 MB 2025-02-15 08:14:07,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22472.78 MB 2025-02-15 08:14:07,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:14:07,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:14:07,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:14:07,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18493.19 MB 2025-02-15 08:14:07,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20382.72 MB 2025-02-15 08:14:07,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:14:07,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 08:14:07,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24266.15 MB 2025-02-15 08:14:07,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:14:07,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21800.15 MB 2025-02-15 08:14:07,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:14:07,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:14:07,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:14:07,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20382.72 MB 2025-02-15 08:14:07,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22624.58 MB 2025-02-15 08:14:07,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:14:07,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24266.15 MB 2025-02-15 08:14:07,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 08:14:07,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:14:07,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28168.86 MB 2025-02-15 08:14:07,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:14:07,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:14:07,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:14:07,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18493.19 MB 2025-02-15 08:14:07,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22624.58 MB 2025-02-15 08:14:07,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:14:07,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 08:14:07,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 08:14:07,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:14:07,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28168.86 MB 2025-02-15 08:14:07,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:14:07,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:14:07,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:14:07,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.12 MB 2025-02-15 08:14:07,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24925.12 MB 2025-02-15 08:14:07,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:14:07,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-15 08:14:07,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:14:07,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:14:07,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25632.91 MB 2025-02-15 08:14:07,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:14:07,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:14:07,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:14:07,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.01 MB 2025-02-15 08:14:07,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25565.97 MB 2025-02-15 08:14:07,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-15 08:14:07,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:14:07,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:14:07,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:14:07,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25791.42 MB 2025-02-15 08:14:07,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:14:07,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:14:07,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.86 seconds 2025-02-15 08:14:07,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14432.02 MB 2025-02-15 08:14:07,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25765.83 MB 2025-02-15 08:14:07,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11333.81 MB 2025-02-15 08:14:07,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53750.01 MB 2025-02-15 08:14:07,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:14:07,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22934.45 MB 2025-02-15 08:14:07,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25791.42 MB 2025-02-15 08:14:07,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:14:07,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:14:07,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 08:14:07,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25765.83 MB 2025-02-15 08:14:07,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19419.54 MB 2025-02-15 08:14:07,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6346.30 MB 2025-02-15 08:14:07,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:14:07,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:14:07,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:14:07,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28262.45 MB 2025-02-15 08:14:07,770 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 08:14:07,771 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:14:07,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:14:07,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:14:07,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:14:07,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:14:07,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19419.54 MB 2025-02-15 08:14:07,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27807.95 MB 2025-02-15 08:14:07,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 08:14:07,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:14:07,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41242.59 MB 2025-02-15 08:14:07,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10427.04 MB 2025-02-15 08:14:07,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27807.95 MB 2025-02-15 08:14:07,937 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 08:14:07,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:14:07,939 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:14:07,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:14:07,940 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:14:07,944 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:14:07,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:14:07,946 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:14:07,946 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:15:01,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:01,404 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:15:01,409 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:15:01,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:01,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1655, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:15:01,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:01,414 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1655, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:15:26,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:15:26,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:15:26,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.49 seconds 2025-02-15 08:15:26,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:26,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24501.01 MB 2025-02-15 08:15:26,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30358.36 MB 2025-02-15 08:15:26,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5857.35 MB 2025-02-15 08:15:26,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53752.10 MB 2025-02-15 08:15:26,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39015.42 MB 2025-02-15 08:15:26,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14736.69 MB 2025-02-15 08:15:26,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39181.71 MB 2025-02-15 08:15:27,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:15:27,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:15:27,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:15:27,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:27,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30358.36 MB 2025-02-15 08:15:27,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24381.68 MB 2025-02-15 08:15:27,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5976.68 MB 2025-02-15 08:15:27,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39015.42 MB 2025-02-15 08:15:27,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50377.79 MB 2025-02-15 08:15:27,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11362.37 MB 2025-02-15 08:15:27,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45320.20 MB 2025-02-15 08:15:28,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:15:28,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:15:28,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:15:28,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:28,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24381.68 MB 2025-02-15 08:15:28,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24912.52 MB 2025-02-15 08:15:28,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:15:28,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50377.79 MB 2025-02-15 08:15:28,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 08:15:28,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15804.14 MB 2025-02-15 08:15:28,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28891.07 MB 2025-02-15 08:15:28,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:15:28,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:15:28,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:15:28,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:28,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-15 08:15:28,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26802.06 MB 2025-02-15 08:15:28,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:15:28,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 08:15:28,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 08:15:28,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:15:28,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28219.48 MB 2025-02-15 08:15:29,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:15:29,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:15:29,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:15:29,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26802.06 MB 2025-02-15 08:15:29,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-15 08:15:29,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:15:29,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 08:15:29,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36932.94 MB 2025-02-15 08:15:29,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 08:15:29,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-15 08:15:29,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:15:29,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:15:29,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:15:29,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-15 08:15:29,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-15 08:15:29,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:15:29,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 08:15:29,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36932.94 MB 2025-02-15 08:15:29,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 08:15:29,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-15 08:15:29,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:15:29,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:15:29,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:15:29,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30577.45 MB 2025-02-15 08:15:29,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31344.46 MB 2025-02-15 08:15:29,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:15:29,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36932.94 MB 2025-02-15 08:15:29,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37348.18 MB 2025-02-15 08:15:29,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:15:29,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32052.24 MB 2025-02-15 08:15:29,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:15:29,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:15:29,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:15:29,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31757.34 MB 2025-02-15 08:15:29,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31987.07 MB 2025-02-15 08:15:29,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.72 MB 2025-02-15 08:15:29,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37348.18 MB 2025-02-15 08:15:29,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37348.18 MB 2025-02-15 08:15:29,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:15:29,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32193.78 MB 2025-02-15 08:15:29,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:15:29,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:15:29,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.95 seconds 2025-02-15 08:15:29,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18734.86 MB 2025-02-15 08:15:29,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32188.09 MB 2025-02-15 08:15:29,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13453.23 MB 2025-02-15 08:15:29,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53752.10 MB 2025-02-15 08:15:29,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37348.18 MB 2025-02-15 08:15:29,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16403.92 MB 2025-02-15 08:15:29,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32193.78 MB 2025-02-15 08:15:29,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:15:29,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:15:29,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:15:29,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32188.09 MB 2025-02-15 08:15:29,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23738.49 MB 2025-02-15 08:15:29,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8449.60 MB 2025-02-15 08:15:29,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37348.18 MB 2025-02-15 08:15:29,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37348.18 MB 2025-02-15 08:15:29,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:15:29,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34699.14 MB 2025-02-15 08:15:29,657 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 08:15:29,657 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:15:29,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:15:29,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:15:29,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:15:29,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:15:29,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23738.49 MB 2025-02-15 08:15:29,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32175.96 MB 2025-02-15 08:15:29,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 08:15:29,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37348.18 MB 2025-02-15 08:15:29,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45736.79 MB 2025-02-15 08:15:29,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 08:15:29,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32175.96 MB 2025-02-15 08:15:29,825 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 08:15:29,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:29,826 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:15:29,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:29,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:15:29,832 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:15:29,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:15:29,833 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:15:29,833 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:16:17,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:17,753 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:16:17,758 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:16:17,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:17,762 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:16:17,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:17,763 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:16:37,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:16:37,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:16:37,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.99 seconds 2025-02-15 08:16:37,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:37,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21957.63 MB 2025-02-15 08:16:37,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26523.13 MB 2025-02-15 08:16:37,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4565.50 MB 2025-02-15 08:16:37,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54125.40 MB 2025-02-15 08:16:37,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37725.67 MB 2025-02-15 08:16:37,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16399.73 MB 2025-02-15 08:16:37,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.87 MB 2025-02-15 08:16:37,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:16:37,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:16:37,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:16:37,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:37,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.13 MB 2025-02-15 08:16:37,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.16 MB 2025-02-15 08:16:37,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4038.97 MB 2025-02-15 08:16:37,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37725.67 MB 2025-02-15 08:16:37,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46762.30 MB 2025-02-15 08:16:37,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9036.63 MB 2025-02-15 08:16:37,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40105.86 MB 2025-02-15 08:16:39,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:16:39,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:16:39,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:16:39,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:39,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.16 MB 2025-02-15 08:16:39,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23015.00 MB 2025-02-15 08:16:39,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:16:39,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46762.30 MB 2025-02-15 08:16:39,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33160.17 MB 2025-02-15 08:16:39,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13602.13 MB 2025-02-15 08:16:39,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.55 MB 2025-02-15 08:16:39,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:16:39,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:16:39,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:16:39,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:39,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-15 08:16:39,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.53 MB 2025-02-15 08:16:39,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:16:39,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33160.17 MB 2025-02-15 08:16:39,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33160.17 MB 2025-02-15 08:16:39,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:16:39,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26321.96 MB 2025-02-15 08:16:39,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:16:39,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:16:39,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:16:39,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:39,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.53 MB 2025-02-15 08:16:39,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-15 08:16:39,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:16:39,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33160.17 MB 2025-02-15 08:16:39,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35047.60 MB 2025-02-15 08:16:39,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:16:39,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-15 08:16:39,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:16:39,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:16:39,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:16:39,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:39,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-15 08:16:39,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-15 08:16:39,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:16:39,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33160.17 MB 2025-02-15 08:16:39,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35047.60 MB 2025-02-15 08:16:39,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:16:39,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-15 08:16:40,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:16:40,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:16:40,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:16:40,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:40,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28679.93 MB 2025-02-15 08:16:40,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29446.93 MB 2025-02-15 08:16:40,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:16:40,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35047.60 MB 2025-02-15 08:16:40,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35460.74 MB 2025-02-15 08:16:40,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 08:16:40,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.72 MB 2025-02-15 08:16:40,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:16:40,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:16:40,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:16:40,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:40,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29859.82 MB 2025-02-15 08:16:40,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30088.16 MB 2025-02-15 08:16:40,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.34 MB 2025-02-15 08:16:40,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35460.74 MB 2025-02-15 08:16:40,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35460.74 MB 2025-02-15 08:16:40,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:16:40,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30328.79 MB 2025-02-15 08:16:40,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:16:40,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:16:40,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.42 seconds 2025-02-15 08:16:40,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:40,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.17 MB 2025-02-15 08:16:40,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30288.15 MB 2025-02-15 08:16:40,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12824.98 MB 2025-02-15 08:16:40,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54125.40 MB 2025-02-15 08:16:40,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35460.74 MB 2025-02-15 08:16:40,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18664.65 MB 2025-02-15 08:16:40,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30328.79 MB 2025-02-15 08:16:40,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:16:40,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:16:40,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:16:40,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:40,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30288.15 MB 2025-02-15 08:16:40,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22451.66 MB 2025-02-15 08:16:40,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7836.49 MB 2025-02-15 08:16:40,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35460.74 MB 2025-02-15 08:16:40,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35460.74 MB 2025-02-15 08:16:40,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:16:40,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32787.16 MB 2025-02-15 08:16:40,474 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-15 08:16:40,474 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:16:40,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:16:40,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:16:40,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:16:40,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:16:40,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22451.66 MB 2025-02-15 08:16:40,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30844.93 MB 2025-02-15 08:16:40,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-15 08:16:40,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35460.74 MB 2025-02-15 08:16:40,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43807.41 MB 2025-02-15 08:16:40,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 08:16:40,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30844.93 MB 2025-02-15 08:16:40,640 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-15 08:16:40,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:40,642 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:16:40,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:40,643 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:16:40,648 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:16:40,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:16:40,649 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:16:40,649 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:17:02,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:02,213 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:17:02,218 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:17:02,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:02,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1023, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:17:02,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:02,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1023, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:17:18,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:17:18,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:17:18,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.95 seconds 2025-02-15 08:17:18,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:18,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20097.13 MB 2025-02-15 08:17:18,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23717.47 MB 2025-02-15 08:17:18,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3620.34 MB 2025-02-15 08:17:18,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52154.07 MB 2025-02-15 08:17:18,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28441.58 MB 2025-02-15 08:17:18,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23712.50 MB 2025-02-15 08:17:18,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32595.64 MB 2025-02-15 08:17:18,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:17:18,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:17:18,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:17:18,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:18,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23717.47 MB 2025-02-15 08:17:18,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21097.16 MB 2025-02-15 08:17:18,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2620.32 MB 2025-02-15 08:17:18,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28441.58 MB 2025-02-15 08:17:18,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36714.84 MB 2025-02-15 08:17:18,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8273.26 MB 2025-02-15 08:17:18,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34089.37 MB 2025-02-15 08:17:20,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:17:20,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:17:20,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:17:20,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21097.16 MB 2025-02-15 08:17:20,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21628.00 MB 2025-02-15 08:17:20,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:17:20,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36714.84 MB 2025-02-15 08:17:20,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26944.21 MB 2025-02-15 08:17:20,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9770.63 MB 2025-02-15 08:17:20,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25606.55 MB 2025-02-15 08:17:20,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:17:20,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:17:20,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:17:20,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21628.00 MB 2025-02-15 08:17:20,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23517.53 MB 2025-02-15 08:17:20,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:17:20,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 08:17:20,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-15 08:17:20,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 08:17:20,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24934.96 MB 2025-02-15 08:17:20,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:17:20,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:17:20,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:17:20,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23517.53 MB 2025-02-15 08:17:20,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25759.39 MB 2025-02-15 08:17:20,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:17:20,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-15 08:17:20,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-15 08:17:20,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 08:17:20,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31303.67 MB 2025-02-15 08:17:20,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:17:20,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:17:20,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:17:20,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21628.00 MB 2025-02-15 08:17:20,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25759.39 MB 2025-02-15 08:17:20,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:17:20,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 08:17:20,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-15 08:17:20,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:17:20,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31303.67 MB 2025-02-15 08:17:20,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:17:20,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:17:20,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:17:20,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27292.93 MB 2025-02-15 08:17:20,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28059.93 MB 2025-02-15 08:17:20,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:17:20,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33078.38 MB 2025-02-15 08:17:20,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-15 08:17:20,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:17:20,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28767.72 MB 2025-02-15 08:17:20,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:17:20,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:17:20,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:17:20,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28472.82 MB 2025-02-15 08:17:20,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28701.49 MB 2025-02-15 08:17:20,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-15 08:17:20,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-15 08:17:20,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-15 08:17:20,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:17:20,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28898.74 MB 2025-02-15 08:17:20,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:17:20,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:17:20,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.37 seconds 2025-02-15 08:17:20,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16532.92 MB 2025-02-15 08:17:20,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28902.07 MB 2025-02-15 08:17:20,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12369.15 MB 2025-02-15 08:17:20,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52154.07 MB 2025-02-15 08:17:20,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-15 08:17:20,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18660.46 MB 2025-02-15 08:17:20,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28902.07 MB 2025-02-15 08:17:20,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:17:20,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:17:20,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:17:20,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28902.07 MB 2025-02-15 08:17:20,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21529.96 MB 2025-02-15 08:17:20,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7372.11 MB 2025-02-15 08:17:20,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-15 08:17:20,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-15 08:17:20,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:17:20,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31407.86 MB 2025-02-15 08:17:20,890 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 08:17:20,890 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:17:20,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:17:20,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:17:20,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:17:20,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:17:20,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21529.96 MB 2025-02-15 08:17:20,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29948.12 MB 2025-02-15 08:17:20,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 08:17:20,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-15 08:17:20,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43956.31 MB 2025-02-15 08:17:20,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-15 08:17:20,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29948.12 MB 2025-02-15 08:17:21,056 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 08:17:21,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:21,058 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:17:21,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:21,059 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:17:21,065 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:17:21,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:17:21,066 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:17:21,066 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:18:38,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:38,480 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:18:38,485 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:18:38,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:38,490 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 468, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:18:38,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:38,492 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 468, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:18:45,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:18:45,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:18:45,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.22 seconds 2025-02-15 08:18:45,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:45,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16229.81 MB 2025-02-15 08:18:45,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17886.03 MB 2025-02-15 08:18:45,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1656.23 MB 2025-02-15 08:18:45,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56509.86 MB 2025-02-15 08:18:45,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20252.20 MB 2025-02-15 08:18:45,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36257.66 MB 2025-02-15 08:18:45,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26833.64 MB 2025-02-15 08:18:45,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:18:45,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:18:45,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 08:18:45,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:45,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17886.03 MB 2025-02-15 08:18:45,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18211.89 MB 2025-02-15 08:18:45,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.85 MB 2025-02-15 08:18:45,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20252.20 MB 2025-02-15 08:18:45,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25490.88 MB 2025-02-15 08:18:45,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5238.69 MB 2025-02-15 08:18:45,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25224.44 MB 2025-02-15 08:18:47,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:18:47,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:18:47,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:18:47,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:47,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18211.89 MB 2025-02-15 08:18:47,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18742.73 MB 2025-02-15 08:18:47,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:18:47,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25490.88 MB 2025-02-15 08:18:47,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 08:18:47,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4057.99 MB 2025-02-15 08:18:47,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22723.35 MB 2025-02-15 08:18:47,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:18:47,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:18:47,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:18:47,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:47,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-15 08:18:47,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20632.26 MB 2025-02-15 08:18:47,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:18:47,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 08:18:47,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24264.05 MB 2025-02-15 08:18:47,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:18:47,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22049.69 MB 2025-02-15 08:18:47,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:18:47,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:18:47,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:18:47,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:47,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20632.26 MB 2025-02-15 08:18:47,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-15 08:18:47,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:18:47,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24264.05 MB 2025-02-15 08:18:47,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 08:18:47,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:18:47,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-15 08:18:47,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:18:47,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:18:47,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:18:47,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:47,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-15 08:18:47,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-15 08:18:47,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:18:47,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 08:18:47,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 08:18:47,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:18:47,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-15 08:18:48,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:18:48,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:18:48,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:18:48,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:48,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24407.66 MB 2025-02-15 08:18:48,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25174.66 MB 2025-02-15 08:18:48,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:18:48,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30398.22 MB 2025-02-15 08:18:48,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:18:48,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:18:48,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25882.45 MB 2025-02-15 08:18:48,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:18:48,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:18:48,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:18:48,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:48,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25587.55 MB 2025-02-15 08:18:48,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25818.94 MB 2025-02-15 08:18:48,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.39 MB 2025-02-15 08:18:48,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:18:48,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:18:48,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:18:48,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26011.31 MB 2025-02-15 08:18:48,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:18:48,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:18:48,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.62 seconds 2025-02-15 08:18:48,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:48,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-15 08:18:48,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26020.02 MB 2025-02-15 08:18:48,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11420.76 MB 2025-02-15 08:18:48,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56509.86 MB 2025-02-15 08:18:48,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:18:48,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25694.31 MB 2025-02-15 08:18:48,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26020.02 MB 2025-02-15 08:18:48,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:18:48,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:18:48,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:18:48,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:48,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26020.02 MB 2025-02-15 08:18:48,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19603.64 MB 2025-02-15 08:18:48,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6416.37 MB 2025-02-15 08:18:48,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:18:48,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:18:48,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:18:48,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28531.68 MB 2025-02-15 08:18:48,396 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:18:48,396 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:18:48,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:18:48,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:18:48,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:18:48,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:18:48,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19603.64 MB 2025-02-15 08:18:48,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28042.67 MB 2025-02-15 08:18:48,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:18:48,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:18:48,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 08:18:48,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:18:48,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28042.67 MB 2025-02-15 08:18:48,564 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:18:48,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:48,565 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:18:48,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:48,566 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:18:48,571 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:18:48,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:18:48,572 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:18:48,572 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:20:31,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:31,084 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:20:31,093 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:20:31,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:31,101 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1614, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:20:31,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:31,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1614, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:20:56,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:20:56,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:20:56,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.91 seconds 2025-02-15 08:20:56,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:56,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24215.32 MB 2025-02-15 08:20:56,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.96 MB 2025-02-15 08:20:56,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5712.64 MB 2025-02-15 08:20:56,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 08:20:56,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38971.38 MB 2025-02-15 08:20:56,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14919.14 MB 2025-02-15 08:20:56,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38896.01 MB 2025-02-15 08:20:56,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:20:56,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:20:56,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:20:56,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:56,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29927.96 MB 2025-02-15 08:20:56,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24168.53 MB 2025-02-15 08:20:56,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5759.43 MB 2025-02-15 08:20:56,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38971.38 MB 2025-02-15 08:20:56,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49838.82 MB 2025-02-15 08:20:56,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10867.44 MB 2025-02-15 08:20:56,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46270.68 MB 2025-02-15 08:20:58,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:20:58,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:20:58,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:20:58,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24168.53 MB 2025-02-15 08:20:58,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24699.37 MB 2025-02-15 08:20:58,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:20:58,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49838.82 MB 2025-02-15 08:20:58,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 08:20:58,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16580.08 MB 2025-02-15 08:20:58,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28677.92 MB 2025-02-15 08:20:58,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:20:58,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:20:58,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:20:58,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-15 08:20:58,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26588.91 MB 2025-02-15 08:20:58,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:20:58,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 08:20:58,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 08:20:58,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:20:58,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28006.34 MB 2025-02-15 08:20:58,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:20:58,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:20:58,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:20:58,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26588.91 MB 2025-02-15 08:20:58,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-15 08:20:58,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:20:58,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 08:20:58,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36561.75 MB 2025-02-15 08:20:58,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 08:20:58,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-15 08:20:58,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:20:58,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:20:58,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:20:58,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-15 08:20:58,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-15 08:20:58,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:20:58,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 08:20:58,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36561.75 MB 2025-02-15 08:20:58,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 08:20:58,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-15 08:20:58,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:20:58,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:20:58,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:20:58,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30364.31 MB 2025-02-15 08:20:58,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31131.31 MB 2025-02-15 08:20:58,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:20:58,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36561.75 MB 2025-02-15 08:20:58,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-15 08:20:58,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:20:58,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.10 MB 2025-02-15 08:20:58,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:20:58,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:20:58,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:20:58,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31544.20 MB 2025-02-15 08:20:58,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31773.53 MB 2025-02-15 08:20:58,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.33 MB 2025-02-15 08:20:58,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-15 08:20:58,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-15 08:20:58,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:20:58,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31990.92 MB 2025-02-15 08:20:58,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:20:58,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:20:58,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.37 seconds 2025-02-15 08:20:58,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18592.01 MB 2025-02-15 08:20:58,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31974.60 MB 2025-02-15 08:20:58,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13382.59 MB 2025-02-15 08:20:58,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 08:20:58,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-15 08:20:58,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16913.53 MB 2025-02-15 08:20:58,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31990.92 MB 2025-02-15 08:20:58,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:20:58,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:20:58,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:20:58,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31974.60 MB 2025-02-15 08:20:58,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23596.40 MB 2025-02-15 08:20:58,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8378.20 MB 2025-02-15 08:20:58,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-15 08:20:58,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-15 08:20:58,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:20:58,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34486.27 MB 2025-02-15 08:20:58,761 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:20:58,761 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:20:58,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:20:58,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:20:58,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:20:58,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:20:58,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23596.40 MB 2025-02-15 08:20:58,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32035.42 MB 2025-02-15 08:20:58,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:20:58,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-15 08:20:58,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41171.29 MB 2025-02-15 08:20:58,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 08:20:58,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32035.42 MB 2025-02-15 08:20:58,925 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:20:58,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:58,927 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:20:58,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:58,928 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:20:58,933 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:20:58,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:20:58,934 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:20:58,934 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:21:19,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:19,933 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:21:19,937 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:21:19,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:19,941 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2253, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:21:19,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:19,942 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2253, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:21:55,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:21:55,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:21:55,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.12 seconds 2025-02-15 08:21:55,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:55,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28667.97 MB 2025-02-15 08:21:55,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36641.34 MB 2025-02-15 08:21:55,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7973.37 MB 2025-02-15 08:21:55,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53756.30 MB 2025-02-15 08:21:55,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41227.91 MB 2025-02-15 08:21:55,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12528.39 MB 2025-02-15 08:21:55,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45613.59 MB 2025-02-15 08:21:55,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:21:55,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:21:55,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:21:55,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:55,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36641.34 MB 2025-02-15 08:21:55,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27491.54 MB 2025-02-15 08:21:55,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9149.80 MB 2025-02-15 08:21:55,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41227.91 MB 2025-02-15 08:21:55,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58068.04 MB 2025-02-15 08:21:55,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16840.13 MB 2025-02-15 08:21:55,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58772.15 MB 2025-02-15 08:21:57,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:21:57,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:21:57,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:21:57,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27491.54 MB 2025-02-15 08:21:57,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28022.38 MB 2025-02-15 08:21:57,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:21:57,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58068.04 MB 2025-02-15 08:21:57,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31184.65 MB 2025-02-15 08:21:57,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26883.39 MB 2025-02-15 08:21:57,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32000.93 MB 2025-02-15 08:21:57,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:21:57,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:21:57,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:21:57,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28022.38 MB 2025-02-15 08:21:57,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29911.92 MB 2025-02-15 08:21:57,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:21:57,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31184.65 MB 2025-02-15 08:21:57,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34015.81 MB 2025-02-15 08:21:57,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:21:57,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31329.35 MB 2025-02-15 08:21:57,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:21:57,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:21:57,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:21:57,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29911.92 MB 2025-02-15 08:21:57,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32153.77 MB 2025-02-15 08:21:57,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:21:57,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34015.81 MB 2025-02-15 08:21:57,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39678.12 MB 2025-02-15 08:21:57,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:21:57,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37698.06 MB 2025-02-15 08:21:57,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:21:57,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:21:57,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:21:57,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28022.38 MB 2025-02-15 08:21:57,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32153.77 MB 2025-02-15 08:21:57,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:21:57,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31184.65 MB 2025-02-15 08:21:57,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39678.12 MB 2025-02-15 08:21:57,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 08:21:57,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37698.06 MB 2025-02-15 08:21:57,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:21:57,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:21:57,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:21:57,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33687.32 MB 2025-02-15 08:21:57,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34454.32 MB 2025-02-15 08:21:57,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:21:57,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39678.12 MB 2025-02-15 08:21:57,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40095.45 MB 2025-02-15 08:21:57,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:21:57,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35162.11 MB 2025-02-15 08:21:57,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:21:57,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:21:57,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:21:57,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34867.21 MB 2025-02-15 08:21:57,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35096.12 MB 2025-02-15 08:21:57,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 08:21:57,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40095.45 MB 2025-02-15 08:21:57,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40095.45 MB 2025-02-15 08:21:57,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:21:57,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35312.59 MB 2025-02-15 08:21:57,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:21:57,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:21:57,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.68 seconds 2025-02-15 08:21:57,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20818.34 MB 2025-02-15 08:21:57,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35296.97 MB 2025-02-15 08:21:57,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14478.63 MB 2025-02-15 08:21:57,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53756.30 MB 2025-02-15 08:21:57,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40095.45 MB 2025-02-15 08:21:57,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13660.85 MB 2025-02-15 08:21:57,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35312.59 MB 2025-02-15 08:21:57,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:21:57,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:21:57,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:21:57,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35296.97 MB 2025-02-15 08:21:57,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25814.31 MB 2025-02-15 08:21:57,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9482.66 MB 2025-02-15 08:21:57,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40095.45 MB 2025-02-15 08:21:57,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40095.45 MB 2025-02-15 08:21:57,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:21:57,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37801.57 MB 2025-02-15 08:21:57,910 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 08:21:57,911 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:21:57,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:21:57,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:21:57,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:21:57,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:21:57,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25814.31 MB 2025-02-15 08:21:57,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34229.26 MB 2025-02-15 08:21:57,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 08:21:57,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40095.45 MB 2025-02-15 08:21:57,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44279.27 MB 2025-02-15 08:21:57,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 08:21:57,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34229.26 MB 2025-02-15 08:21:58,074 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 08:21:58,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:58,076 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:21:58,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:58,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:21:58,081 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:21:58,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:21:58,082 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:21:58,082 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:22:45,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:45,055 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:22:45,060 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:22:45,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:45,064 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 464, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:22:45,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:45,065 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 464, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:22:52,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:22:52,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:22:52,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.22 seconds 2025-02-15 08:22:52,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:52,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16201.93 MB 2025-02-15 08:22:52,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17844.00 MB 2025-02-15 08:22:52,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.07 MB 2025-02-15 08:22:52,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52646.90 MB 2025-02-15 08:22:52,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20254.29 MB 2025-02-15 08:22:52,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32392.61 MB 2025-02-15 08:22:52,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26805.77 MB 2025-02-15 08:22:52,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:22:52,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:22:52,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 08:22:52,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:52,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.00 MB 2025-02-15 08:22:52,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18191.09 MB 2025-02-15 08:22:52,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.09 MB 2025-02-15 08:22:52,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20254.29 MB 2025-02-15 08:22:52,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25639.78 MB 2025-02-15 08:22:52,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5385.49 MB 2025-02-15 08:22:52,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25329.57 MB 2025-02-15 08:22:54,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:22:54,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:22:54,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:22:54,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18191.09 MB 2025-02-15 08:22:54,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18721.93 MB 2025-02-15 08:22:54,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:22:54,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25639.78 MB 2025-02-15 08:22:54,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21434.99 MB 2025-02-15 08:22:54,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4204.79 MB 2025-02-15 08:22:54,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22702.56 MB 2025-02-15 08:22:54,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:22:54,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:22:54,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:22:54,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18721.93 MB 2025-02-15 08:22:54,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20611.47 MB 2025-02-15 08:22:54,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:22:54,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 08:22:54,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24266.15 MB 2025-02-15 08:22:54,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:22:54,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22028.90 MB 2025-02-15 08:22:54,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:22:54,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:22:54,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:22:54,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20611.47 MB 2025-02-15 08:22:54,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.32 MB 2025-02-15 08:22:54,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:22:54,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24266.15 MB 2025-02-15 08:22:54,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 08:22:54,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:22:54,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.60 MB 2025-02-15 08:22:54,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:22:54,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:22:54,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:22:54,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18721.93 MB 2025-02-15 08:22:54,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.32 MB 2025-02-15 08:22:54,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:22:54,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 08:22:54,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 08:22:54,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:22:54,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.60 MB 2025-02-15 08:22:54,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:22:54,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:22:54,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:22:54,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24386.86 MB 2025-02-15 08:22:54,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25153.87 MB 2025-02-15 08:22:54,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:22:54,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-15 08:22:54,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:22:54,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:22:54,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25861.66 MB 2025-02-15 08:22:54,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:22:54,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:22:54,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:22:54,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25566.76 MB 2025-02-15 08:22:54,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25795.85 MB 2025-02-15 08:22:54,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-15 08:22:54,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:22:54,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:22:54,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:22:54,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25963.94 MB 2025-02-15 08:22:54,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:22:54,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:22:54,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.63 seconds 2025-02-15 08:22:54,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14585.32 MB 2025-02-15 08:22:54,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25996.92 MB 2025-02-15 08:22:54,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11411.60 MB 2025-02-15 08:22:54,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52646.90 MB 2025-02-15 08:22:54,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:22:54,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21831.35 MB 2025-02-15 08:22:54,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25996.92 MB 2025-02-15 08:22:54,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:22:54,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:22:54,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:22:54,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25996.92 MB 2025-02-15 08:22:54,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19589.71 MB 2025-02-15 08:22:54,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6407.22 MB 2025-02-15 08:22:54,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:22:54,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:22:54,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:22:54,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28508.59 MB 2025-02-15 08:22:54,990 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:22:54,991 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:22:54,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:22:54,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:22:54,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:22:54,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:22:54,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19589.71 MB 2025-02-15 08:22:54,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28028.73 MB 2025-02-15 08:22:54,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:22:54,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:22:54,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 08:22:54,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:22:54,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28028.73 MB 2025-02-15 08:22:55,156 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:22:55,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:55,157 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:22:55,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:55,158 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:22:55,163 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:22:55,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:22:55,164 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:22:55,164 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:23:19,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:19,128 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:23:19,136 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:23:19,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:19,142 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1032, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:23:19,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:19,144 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1032, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:23:35,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:23:35,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:23:35,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.19 seconds 2025-02-15 08:23:35,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:35,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20159.85 MB 2025-02-15 08:23:35,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23813.09 MB 2025-02-15 08:23:35,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3653.24 MB 2025-02-15 08:23:35,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 08:23:35,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26424.12 MB 2025-02-15 08:23:35,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27466.40 MB 2025-02-15 08:23:35,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32802.92 MB 2025-02-15 08:23:35,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:23:35,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:23:35,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:23:35,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:35,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23813.09 MB 2025-02-15 08:23:35,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21143.95 MB 2025-02-15 08:23:35,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2669.14 MB 2025-02-15 08:23:35,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26424.12 MB 2025-02-15 08:23:35,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35399.93 MB 2025-02-15 08:23:35,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8975.81 MB 2025-02-15 08:23:35,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34887.09 MB 2025-02-15 08:23:37,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:23:37,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:23:37,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:23:37,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21143.95 MB 2025-02-15 08:23:37,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21674.79 MB 2025-02-15 08:23:37,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:23:37,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35399.93 MB 2025-02-15 08:23:37,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24895.29 MB 2025-02-15 08:23:37,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10504.63 MB 2025-02-15 08:23:37,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25654.37 MB 2025-02-15 08:23:37,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:23:37,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:23:37,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:23:37,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21674.79 MB 2025-02-15 08:23:37,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23564.32 MB 2025-02-15 08:23:37,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:23:37,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 08:23:37,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26782.73 MB 2025-02-15 08:23:37,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:23:37,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24981.75 MB 2025-02-15 08:23:37,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:23:37,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:23:37,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:23:37,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23564.32 MB 2025-02-15 08:23:37,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25806.18 MB 2025-02-15 08:23:37,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:23:37,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26782.73 MB 2025-02-15 08:23:37,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 08:23:37,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:23:37,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31350.46 MB 2025-02-15 08:23:37,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:23:37,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:23:37,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:23:37,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21674.79 MB 2025-02-15 08:23:37,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25806.18 MB 2025-02-15 08:23:37,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:23:37,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 08:23:37,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 08:23:37,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 08:23:37,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31350.46 MB 2025-02-15 08:23:37,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:23:37,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:23:37,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:23:37,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27339.72 MB 2025-02-15 08:23:37,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28106.72 MB 2025-02-15 08:23:37,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:23:37,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33388.76 MB 2025-02-15 08:23:37,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:23:37,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:23:37,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28814.51 MB 2025-02-15 08:23:37,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:23:37,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:23:37,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:23:37,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28519.61 MB 2025-02-15 08:23:37,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28747.66 MB 2025-02-15 08:23:37,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-15 08:23:37,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:23:37,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:23:37,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:23:37,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28954.06 MB 2025-02-15 08:23:37,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:23:37,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:23:37,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.65 seconds 2025-02-15 08:23:37,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:37,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16564.28 MB 2025-02-15 08:23:37,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28948.51 MB 2025-02-15 08:23:37,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12384.24 MB 2025-02-15 08:23:37,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 08:23:37,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:23:37,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20084.42 MB 2025-02-15 08:23:37,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28954.06 MB 2025-02-15 08:23:38,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:23:38,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:23:38,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:23:38,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:38,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28948.51 MB 2025-02-15 08:23:38,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21557.04 MB 2025-02-15 08:23:38,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7391.47 MB 2025-02-15 08:23:38,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:23:38,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:23:38,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:23:38,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31450.35 MB 2025-02-15 08:23:38,082 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 08:23:38,082 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 08:23:38,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:23:38,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:23:38,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:23:38,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:23:38,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21557.04 MB 2025-02-15 08:23:38,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29962.70 MB 2025-02-15 08:23:38,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 08:23:38,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:23:38,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42165.34 MB 2025-02-15 08:23:38,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 08:23:38,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29962.70 MB 2025-02-15 08:23:38,246 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 08:23:38,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:38,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:23:38,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:38,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:23:38,253 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:23:38,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:38,254 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:23:38,255 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 08:23:59,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:59,418 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:23:59,423 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:23:59,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:59,426 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:23:59,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:23:59,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:24:06,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:24:06,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:24:06,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.36 seconds 2025-02-15 08:24:06,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:06,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.65 MB 2025-02-15 08:24:06,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.57 MB 2025-02-15 08:24:06,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.92 MB 2025-02-15 08:24:06,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50524.59 MB 2025-02-15 08:24:06,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20252.20 MB 2025-02-15 08:24:06,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30272.39 MB 2025-02-15 08:24:06,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26868.48 MB 2025-02-15 08:24:06,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:24:06,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:24:06,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 08:24:06,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:06,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.57 MB 2025-02-15 08:24:06,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18237.88 MB 2025-02-15 08:24:06,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.31 MB 2025-02-15 08:24:06,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20252.20 MB 2025-02-15 08:24:06,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25623.00 MB 2025-02-15 08:24:06,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5370.81 MB 2025-02-15 08:24:06,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25410.79 MB 2025-02-15 08:24:08,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:24:08,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:24:08,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:24:08,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:08,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18237.88 MB 2025-02-15 08:24:08,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18768.72 MB 2025-02-15 08:24:08,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:24:08,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25623.00 MB 2025-02-15 08:24:08,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 08:24:08,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4190.11 MB 2025-02-15 08:24:08,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22748.31 MB 2025-02-15 08:24:08,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:24:08,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:24:08,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:24:08,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:08,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-15 08:24:08,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20658.25 MB 2025-02-15 08:24:08,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:24:08,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 08:24:08,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24264.05 MB 2025-02-15 08:24:08,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:24:08,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22075.68 MB 2025-02-15 08:24:09,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:24:09,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:24:09,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:24:09,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20658.25 MB 2025-02-15 08:24:09,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-15 08:24:09,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:24:09,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24264.05 MB 2025-02-15 08:24:09,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 08:24:09,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:24:09,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-15 08:24:09,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:24:09,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:24:09,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:24:09,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-15 08:24:09,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-15 08:24:09,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:24:09,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 08:24:09,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 08:24:09,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:24:09,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-15 08:24:09,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:24:09,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:24:09,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:24:09,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24433.65 MB 2025-02-15 08:24:09,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25200.65 MB 2025-02-15 08:24:09,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:24:09,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30398.22 MB 2025-02-15 08:24:09,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:24:09,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:24:09,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.44 MB 2025-02-15 08:24:09,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:24:09,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:24:09,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:24:09,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25613.54 MB 2025-02-15 08:24:09,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25841.76 MB 2025-02-15 08:24:09,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.21 MB 2025-02-15 08:24:09,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:24:09,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:24:09,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:24:09,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26022.28 MB 2025-02-15 08:24:09,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:24:09,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:24:09,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.80 seconds 2025-02-15 08:24:09,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14616.68 MB 2025-02-15 08:24:09,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26042.61 MB 2025-02-15 08:24:09,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11425.93 MB 2025-02-15 08:24:09,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50524.59 MB 2025-02-15 08:24:09,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:24:09,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19709.03 MB 2025-02-15 08:24:09,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26042.61 MB 2025-02-15 08:24:09,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:24:09,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:24:09,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:24:09,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26042.61 MB 2025-02-15 08:24:09,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19614.79 MB 2025-02-15 08:24:09,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6427.82 MB 2025-02-15 08:24:09,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:24:09,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 08:24:09,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:24:09,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28549.05 MB 2025-02-15 08:24:09,519 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 08:24:09,520 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:24:09,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:24:09,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:24:09,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:24:09,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:24:09,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19614.79 MB 2025-02-15 08:24:09,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28036.75 MB 2025-02-15 08:24:09,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-15 08:24:09,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 08:24:09,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41280.34 MB 2025-02-15 08:24:09,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 08:24:09,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28036.75 MB 2025-02-15 08:24:09,686 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 08:24:09,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:24:09,687 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:24:09,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:24:09,688 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:24:09,693 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:24:09,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:24:09,694 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:24:09,694 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:25:53,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:25:53,979 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:25:53,984 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:25:53,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:25:53,988 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 364, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:25:53,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:25:53,989 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 364, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:25:59,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:25:59,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:25:59,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.62 seconds 2025-02-15 08:25:59,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:25:59,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15505.12 MB 2025-02-15 08:25:59,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16793.29 MB 2025-02-15 08:25:59,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1288.18 MB 2025-02-15 08:25:59,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49652.17 MB 2025-02-15 08:25:59,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19073.60 MB 2025-02-15 08:25:59,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30578.57 MB 2025-02-15 08:25:59,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25655.97 MB 2025-02-15 08:25:59,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:25:59,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:25:59,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:25:59,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:25:59,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16793.29 MB 2025-02-15 08:25:59,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17192.61 MB 2025-02-15 08:25:59,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 399.32 MB 2025-02-15 08:25:59,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19073.60 MB 2025-02-15 08:25:59,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23156.75 MB 2025-02-15 08:25:59,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4083.15 MB 2025-02-15 08:25:59,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21456.58 MB 2025-02-15 08:26:01,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:26:01,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:26:01,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.59 seconds 2025-02-15 08:26:01,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17192.61 MB 2025-02-15 08:26:01,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.21 MB 2025-02-15 08:26:01,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-15 08:26:01,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23156.75 MB 2025-02-15 08:26:01,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19876.81 MB 2025-02-15 08:26:01,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3279.95 MB 2025-02-15 08:26:01,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21618.10 MB 2025-02-15 08:26:01,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:26:01,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:26:01,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:26:01,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17633.21 MB 2025-02-15 08:26:01,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19203.45 MB 2025-02-15 08:26:01,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1570.24 MB 2025-02-15 08:26:01,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19876.81 MB 2025-02-15 08:26:01,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22229.81 MB 2025-02-15 08:26:01,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2353.00 MB 2025-02-15 08:26:01,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20380.97 MB 2025-02-15 08:26:01,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:26:01,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:26:01,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 08:26:01,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19203.45 MB 2025-02-15 08:26:01,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21065.25 MB 2025-02-15 08:26:01,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1861.80 MB 2025-02-15 08:26:01,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22229.81 MB 2025-02-15 08:26:01,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27327.99 MB 2025-02-15 08:26:01,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5098.18 MB 2025-02-15 08:26:01,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25672.76 MB 2025-02-15 08:26:01,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:26:01,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:26:01,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 08:26:01,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17633.21 MB 2025-02-15 08:26:01,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21065.25 MB 2025-02-15 08:26:01,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3432.04 MB 2025-02-15 08:26:01,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19876.81 MB 2025-02-15 08:26:01,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27327.99 MB 2025-02-15 08:26:01,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7451.18 MB 2025-02-15 08:26:01,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25672.76 MB 2025-02-15 08:26:01,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:26:01,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:26:01,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 08:26:01,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22338.61 MB 2025-02-15 08:26:01,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22976.27 MB 2025-02-15 08:26:01,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.66 MB 2025-02-15 08:26:01,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27327.99 MB 2025-02-15 08:26:01,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27674.02 MB 2025-02-15 08:26:01,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 346.03 MB 2025-02-15 08:26:01,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23563.74 MB 2025-02-15 08:26:01,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:26:01,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:26:01,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:26:01,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23318.97 MB 2025-02-15 08:26:01,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23532.24 MB 2025-02-15 08:26:01,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.26 MB 2025-02-15 08:26:01,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27674.02 MB 2025-02-15 08:26:01,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27674.02 MB 2025-02-15 08:26:01,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:26:01,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23670.90 MB 2025-02-15 08:26:01,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:26:01,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:26:01,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.60 seconds 2025-02-15 08:26:01,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-15 08:26:01,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23732.45 MB 2025-02-15 08:26:01,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9495.54 MB 2025-02-15 08:26:01,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49652.17 MB 2025-02-15 08:26:01,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27674.02 MB 2025-02-15 08:26:01,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21978.15 MB 2025-02-15 08:26:01,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23732.45 MB 2025-02-15 08:26:01,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:26:01,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:26:01,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:26:01,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23732.45 MB 2025-02-15 08:26:01,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26733.58 MB 2025-02-15 08:26:01,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3001.13 MB 2025-02-15 08:26:01,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27674.02 MB 2025-02-15 08:26:01,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28210.89 MB 2025-02-15 08:26:01,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 536.87 MB 2025-02-15 08:26:01,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27034.50 MB 2025-02-15 08:26:01,879 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-15 08:26:01,879 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:26:01,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:26:01,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:26:01,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:26:01,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:26:01,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18908.63 MB 2025-02-15 08:26:01,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27312.19 MB 2025-02-15 08:26:01,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-15 08:26:01,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28210.89 MB 2025-02-15 08:26:01,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36565.94 MB 2025-02-15 08:26:01,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 08:26:01,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27312.19 MB 2025-02-15 08:26:02,047 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-15 08:26:02,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:26:02,048 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:26:02,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:26:02,049 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:26:02,054 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:26:02,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:26:02,055 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:26:02,055 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:27:31,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:27:31,593 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:27:31,597 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:27:31,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:27:31,601 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2781, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:27:31,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:27:31,602 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2781, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:28:14,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:28:14,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:28:14,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.83 seconds 2025-02-15 08:28:14,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:14,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32348.36 MB 2025-02-15 08:28:14,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42190.29 MB 2025-02-15 08:28:14,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9841.93 MB 2025-02-15 08:28:14,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64302.87 MB 2025-02-15 08:28:14,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45701.14 MB 2025-02-15 08:28:14,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18601.74 MB 2025-02-15 08:28:14,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52032.09 MB 2025-02-15 08:28:14,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:28:14,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:28:14,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:28:14,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:14,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42190.29 MB 2025-02-15 08:28:14,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30237.65 MB 2025-02-15 08:28:14,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11952.64 MB 2025-02-15 08:28:14,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45701.14 MB 2025-02-15 08:28:14,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67358.43 MB 2025-02-15 08:28:14,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21657.29 MB 2025-02-15 08:28:14,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 71564.07 MB 2025-02-15 08:28:16,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:28:16,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:28:16,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:28:16,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:16,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30237.65 MB 2025-02-15 08:28:16,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.49 MB 2025-02-15 08:28:16,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:28:16,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67358.43 MB 2025-02-15 08:28:16,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33061.60 MB 2025-02-15 08:28:16,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34296.82 MB 2025-02-15 08:28:16,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34747.04 MB 2025-02-15 08:28:16,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:28:16,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:28:16,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:28:16,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:16,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30768.49 MB 2025-02-15 08:28:16,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32658.02 MB 2025-02-15 08:28:16,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:28:16,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33061.60 MB 2025-02-15 08:28:16,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35892.76 MB 2025-02-15 08:28:16,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:28:16,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34075.45 MB 2025-02-15 08:28:16,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:28:16,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:28:16,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:28:16,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:16,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32658.02 MB 2025-02-15 08:28:16,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34899.88 MB 2025-02-15 08:28:16,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:28:16,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35892.76 MB 2025-02-15 08:28:16,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42026.93 MB 2025-02-15 08:28:16,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:28:16,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40444.16 MB 2025-02-15 08:28:16,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:28:16,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:28:16,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:28:16,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:16,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30768.49 MB 2025-02-15 08:28:16,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34899.88 MB 2025-02-15 08:28:16,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:28:16,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33061.60 MB 2025-02-15 08:28:16,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42026.93 MB 2025-02-15 08:28:16,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:28:16,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40444.16 MB 2025-02-15 08:28:17,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:28:17,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:28:17,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:28:17,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:17,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36433.42 MB 2025-02-15 08:28:17,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37200.42 MB 2025-02-15 08:28:17,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:28:17,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42026.93 MB 2025-02-15 08:28:17,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42444.26 MB 2025-02-15 08:28:17,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:28:17,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37908.21 MB 2025-02-15 08:28:17,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:28:17,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:28:17,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:28:17,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:17,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37613.31 MB 2025-02-15 08:28:17,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37842.03 MB 2025-02-15 08:28:17,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-15 08:28:17,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42444.26 MB 2025-02-15 08:28:17,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42444.26 MB 2025-02-15 08:28:17,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:28:17,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38083.62 MB 2025-02-15 08:28:17,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:28:17,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:28:17,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.47 seconds 2025-02-15 08:28:17,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:17,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22658.53 MB 2025-02-15 08:28:17,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38042.66 MB 2025-02-15 08:28:17,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15384.13 MB 2025-02-15 08:28:17,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54611.94 MB 2025-02-15 08:28:17,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42444.26 MB 2025-02-15 08:28:17,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12167.68 MB 2025-02-15 08:28:17,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38083.62 MB 2025-02-15 08:28:17,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:28:17,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:28:17,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:28:17,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:17,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38042.66 MB 2025-02-15 08:28:17,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27656.29 MB 2025-02-15 08:28:17,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10386.37 MB 2025-02-15 08:28:17,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42444.26 MB 2025-02-15 08:28:17,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42444.26 MB 2025-02-15 08:28:17,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:28:17,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40548.80 MB 2025-02-15 08:28:17,362 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 08:28:17,363 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:28:17,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:28:17,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:28:17,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:28:17,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:28:17,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27656.29 MB 2025-02-15 08:28:17,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36077.06 MB 2025-02-15 08:28:17,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 08:28:17,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42444.26 MB 2025-02-15 08:28:17,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46630.17 MB 2025-02-15 08:28:17,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 08:28:17,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36077.06 MB 2025-02-15 08:28:17,527 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 08:28:17,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:28:17,528 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:28:17,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:28:17,529 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:28:17,534 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:28:17,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:28:17,535 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:28:17,535 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:29:48,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:29:48,033 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:29:48,041 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:29:48,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:29:48,047 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:29:48,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:29:48,049 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:30:16,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:30:16,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:30:16,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.96 seconds 2025-02-15 08:30:16,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:16,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25567.14 MB 2025-02-15 08:30:16,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31965.49 MB 2025-02-15 08:30:16,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6398.35 MB 2025-02-15 08:30:16,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55002.01 MB 2025-02-15 08:30:16,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40204.50 MB 2025-02-15 08:30:16,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14797.50 MB 2025-02-15 08:30:16,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40927.25 MB 2025-02-15 08:30:16,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:30:16,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:30:16,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 08:30:16,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:16,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31965.49 MB 2025-02-15 08:30:16,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25177.08 MB 2025-02-15 08:30:16,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6788.41 MB 2025-02-15 08:30:16,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40204.50 MB 2025-02-15 08:30:16,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52021.95 MB 2025-02-15 08:30:16,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11817.45 MB 2025-02-15 08:30:16,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49730.84 MB 2025-02-15 08:30:18,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:30:18,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:30:18,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:30:18,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25177.08 MB 2025-02-15 08:30:18,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25707.92 MB 2025-02-15 08:30:18,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:30:18,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52021.95 MB 2025-02-15 08:30:18,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:30:18,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18215.86 MB 2025-02-15 08:30:18,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.47 MB 2025-02-15 08:30:18,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:30:18,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:30:18,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:30:18,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25707.92 MB 2025-02-15 08:30:18,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27597.45 MB 2025-02-15 08:30:18,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:30:18,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:30:18,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 08:30:18,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:30:18,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29014.88 MB 2025-02-15 08:30:18,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:30:18,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:30:18,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:30:18,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27597.45 MB 2025-02-15 08:30:18,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29839.31 MB 2025-02-15 08:30:18,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:30:18,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:30:18,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37580.96 MB 2025-02-15 08:30:18,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:30:18,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35383.59 MB 2025-02-15 08:30:18,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:30:18,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:30:18,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:30:18,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25707.92 MB 2025-02-15 08:30:18,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29839.31 MB 2025-02-15 08:30:18,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:30:18,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 08:30:18,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37580.96 MB 2025-02-15 08:30:18,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:30:18,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35383.59 MB 2025-02-15 08:30:18,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:30:18,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:30:18,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:30:18,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31372.85 MB 2025-02-15 08:30:18,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32139.85 MB 2025-02-15 08:30:18,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:30:18,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37580.96 MB 2025-02-15 08:30:18,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37996.20 MB 2025-02-15 08:30:18,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:30:18,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32847.64 MB 2025-02-15 08:30:18,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:30:18,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:30:18,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:30:18,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32552.74 MB 2025-02-15 08:30:18,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32781.88 MB 2025-02-15 08:30:18,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-15 08:30:18,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37996.20 MB 2025-02-15 08:30:18,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37996.20 MB 2025-02-15 08:30:18,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:30:18,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33003.50 MB 2025-02-15 08:30:18,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:30:18,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:30:18,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.43 seconds 2025-02-15 08:30:18,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19267.92 MB 2025-02-15 08:30:18,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32982.92 MB 2025-02-15 08:30:18,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13715.00 MB 2025-02-15 08:30:18,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55002.01 MB 2025-02-15 08:30:18,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37996.20 MB 2025-02-15 08:30:18,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17005.81 MB 2025-02-15 08:30:18,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33003.50 MB 2025-02-15 08:30:18,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:30:18,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:30:18,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:30:18,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21258.27 MB 2025-02-15 08:30:18,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24271.93 MB 2025-02-15 08:30:18,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3013.66 MB 2025-02-15 08:30:18,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37996.20 MB 2025-02-15 08:30:18,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37996.20 MB 2025-02-15 08:30:18,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:30:18,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24573.26 MB 2025-02-15 08:30:18,775 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 08:30:18,775 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:30:18,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:30:18,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:30:18,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:30:18,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:30:18,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24271.93 MB 2025-02-15 08:30:18,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32710.77 MB 2025-02-15 08:30:18,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 08:30:18,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37996.20 MB 2025-02-15 08:30:18,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46384.81 MB 2025-02-15 08:30:18,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 08:30:18,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32710.77 MB 2025-02-15 08:30:18,940 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 08:30:18,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:18,941 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:30:18,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:18,942 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:30:18,947 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:30:18,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:18,948 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:30:18,948 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:30:28,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:28,751 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:30:28,755 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:30:28,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:28,759 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2218, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:30:28,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:30:28,760 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2218, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:31:03,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:31:03,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:31:03,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.65 seconds 2025-02-15 08:31:03,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:03,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28424.09 MB 2025-02-15 08:31:03,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36273.73 MB 2025-02-15 08:31:03,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7849.64 MB 2025-02-15 08:31:03,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54773.42 MB 2025-02-15 08:31:03,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41655.73 MB 2025-02-15 08:31:03,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13117.69 MB 2025-02-15 08:31:03,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45143.21 MB 2025-02-15 08:31:03,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:31:03,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:31:03,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 08:31:03,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:03,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36273.73 MB 2025-02-15 08:31:03,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27309.59 MB 2025-02-15 08:31:03,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8964.14 MB 2025-02-15 08:31:03,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41655.73 MB 2025-02-15 08:31:03,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51753.52 MB 2025-02-15 08:31:03,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10097.79 MB 2025-02-15 08:31:03,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48471.49 MB 2025-02-15 08:31:05,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:31:05,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:31:05,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:31:05,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27309.59 MB 2025-02-15 08:31:05,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27840.43 MB 2025-02-15 08:31:05,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:31:05,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51753.52 MB 2025-02-15 08:31:05,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31744.59 MB 2025-02-15 08:31:05,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20008.93 MB 2025-02-15 08:31:05,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31820.02 MB 2025-02-15 08:31:05,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:31:05,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:31:05,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:31:05,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27840.43 MB 2025-02-15 08:31:05,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29729.97 MB 2025-02-15 08:31:05,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:31:05,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31744.59 MB 2025-02-15 08:31:05,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33632.03 MB 2025-02-15 08:31:05,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:31:05,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31147.39 MB 2025-02-15 08:31:05,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:31:05,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:31:05,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:31:05,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.97 MB 2025-02-15 08:31:05,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31971.82 MB 2025-02-15 08:31:05,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:31:05,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33632.03 MB 2025-02-15 08:31:05,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 08:31:05,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:31:05,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37516.10 MB 2025-02-15 08:31:05,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:31:05,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:31:05,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:31:05,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27840.43 MB 2025-02-15 08:31:05,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31971.82 MB 2025-02-15 08:31:05,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:31:05,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31744.59 MB 2025-02-15 08:31:05,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39766.20 MB 2025-02-15 08:31:05,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 08:31:05,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37516.10 MB 2025-02-15 08:31:05,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:31:05,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:31:05,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:31:05,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33505.36 MB 2025-02-15 08:31:05,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34272.37 MB 2025-02-15 08:31:05,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:31:05,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39766.20 MB 2025-02-15 08:31:05,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 08:31:05,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:31:05,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34980.15 MB 2025-02-15 08:31:05,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:31:05,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:31:05,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:31:05,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34685.25 MB 2025-02-15 08:31:05,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34911.62 MB 2025-02-15 08:31:05,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.36 MB 2025-02-15 08:31:05,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40181.43 MB 2025-02-15 08:31:05,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 08:31:05,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:31:05,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35132.72 MB 2025-02-15 08:31:05,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:31:05,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:31:05,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.15 seconds 2025-02-15 08:31:05,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:05,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20696.40 MB 2025-02-15 08:31:05,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35112.69 MB 2025-02-15 08:31:05,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14416.30 MB 2025-02-15 08:31:05,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54773.42 MB 2025-02-15 08:31:05,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 08:31:05,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14591.98 MB 2025-02-15 08:31:05,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35132.72 MB 2025-02-15 08:31:06,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:31:06,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:31:06,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:31:06,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:06,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35112.69 MB 2025-02-15 08:31:06,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25700.79 MB 2025-02-15 08:31:06,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9411.91 MB 2025-02-15 08:31:06,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40181.43 MB 2025-02-15 08:31:06,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 08:31:06,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:31:06,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37624.36 MB 2025-02-15 08:31:06,195 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:31:06,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:31:06,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:31:06,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:31:06,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:31:06,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:31:06,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25700.79 MB 2025-02-15 08:31:06,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34139.81 MB 2025-02-15 08:31:06,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:31:06,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40181.43 MB 2025-02-15 08:31:06,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48572.14 MB 2025-02-15 08:31:06,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 08:31:06,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34139.81 MB 2025-02-15 08:31:06,359 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:31:06,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:06,360 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:31:06,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:06,361 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:31:06,366 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:31:06,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:06,367 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:31:06,367 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:31:58,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:58,631 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:31:58,636 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:31:58,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:58,639 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:31:58,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:31:58,640 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:32:01,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:32:01,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:32:01,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.52 seconds 2025-02-15 08:32:01,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:01,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-15 08:32:01,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-15 08:32:01,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 08:32:01,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61157.15 MB 2025-02-15 08:32:01,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18605.93 MB 2025-02-15 08:32:01,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42551.21 MB 2025-02-15 08:32:01,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-15 08:32:01,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:32:01,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:32:01,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:32:01,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:01,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-15 08:32:01,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-15 08:32:01,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-15 08:32:01,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 08:32:01,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18605.93 MB 2025-02-15 08:32:01,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:01,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16921.83 MB 2025-02-15 08:32:01,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:32:01,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:32:01,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-15 08:32:01,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:01,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-15 08:32:01,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-15 08:32:01,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 08:32:01,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 08:32:01,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18605.93 MB 2025-02-15 08:32:01,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:01,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.09 MB 2025-02-15 08:32:01,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:32:01,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:32:01,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:32:01,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:01,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 08:32:01,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-15 08:32:01,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 08:32:01,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 08:32:01,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18605.93 MB 2025-02-15 08:32:01,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:01,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-15 08:32:02,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:32:02,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:32:02,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:32:02,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-15 08:32:02,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 08:32:02,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 08:32:02,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 08:32:02,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20323.50 MB 2025-02-15 08:32:02,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-15 08:32:02,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19045.19 MB 2025-02-15 08:32:02,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:32:02,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:32:02,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:32:02,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 08:32:02,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 08:32:02,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 08:32:02,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 08:32:02,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20323.50 MB 2025-02-15 08:32:02,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-15 08:32:02,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19045.19 MB 2025-02-15 08:32:02,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:32:02,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:32:02,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 08:32:02,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-15 08:32:02,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.63 MB 2025-02-15 08:32:02,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-15 08:32:02,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20323.50 MB 2025-02-15 08:32:02,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20484.98 MB 2025-02-15 08:32:02,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-15 08:32:02,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18032.36 MB 2025-02-15 08:32:02,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:32:02,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:32:02,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:32:02,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17905.82 MB 2025-02-15 08:32:02,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18135.07 MB 2025-02-15 08:32:02,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.25 MB 2025-02-15 08:32:02,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20484.98 MB 2025-02-15 08:32:02,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20484.98 MB 2025-02-15 08:32:02,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:02,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18159.63 MB 2025-02-15 08:32:02,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:32:02,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:32:02,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-15 08:32:02,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-15 08:32:02,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18336.14 MB 2025-02-15 08:32:02,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4806.50 MB 2025-02-15 08:32:02,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61157.15 MB 2025-02-15 08:32:02,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20484.98 MB 2025-02-15 08:32:02,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40672.17 MB 2025-02-15 08:32:02,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18336.14 MB 2025-02-15 08:32:02,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:32:02,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:32:02,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:32:02,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18336.14 MB 2025-02-15 08:32:02,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17407.04 MB 2025-02-15 08:32:02,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -929.11 MB 2025-02-15 08:32:02,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20484.98 MB 2025-02-15 08:32:02,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20484.98 MB 2025-02-15 08:32:02,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:02,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19139.88 MB 2025-02-15 08:32:02,408 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:32:02,408 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 08:32:02,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:32:02,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:32:02,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:32:02,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:02,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17407.04 MB 2025-02-15 08:32:02,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25846.06 MB 2025-02-15 08:32:02,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:32:02,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20484.98 MB 2025-02-15 08:32:02,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30974.94 MB 2025-02-15 08:32:02,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:32:02,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25846.06 MB 2025-02-15 08:32:02,573 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:32:02,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:02,574 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:32:02,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:02,575 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:32:02,580 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:32:02,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:02,581 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:32:02,581 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 08:32:11,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:11,791 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:32:11,796 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:32:11,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:11,799 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:32:11,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:11,800 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:32:30,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:32:30,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:32:30,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.30 seconds 2025-02-15 08:32:30,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:30,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21198.10 MB 2025-02-15 08:32:30,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25377.73 MB 2025-02-15 08:32:30,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-15 08:32:30,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43559.94 MB 2025-02-15 08:32:30,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26952.60 MB 2025-02-15 08:32:30,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16607.35 MB 2025-02-15 08:32:30,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34295.04 MB 2025-02-15 08:32:30,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:32:30,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:32:30,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 08:32:30,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:30,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25377.73 MB 2025-02-15 08:32:30,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21918.55 MB 2025-02-15 08:32:30,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3459.18 MB 2025-02-15 08:32:30,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26952.60 MB 2025-02-15 08:32:30,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37289.46 MB 2025-02-15 08:32:30,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10336.86 MB 2025-02-15 08:32:30,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37814.63 MB 2025-02-15 08:32:32,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:32:32,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:32:32,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:32:32,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21918.55 MB 2025-02-15 08:32:32,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22449.39 MB 2025-02-15 08:32:32,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:32:32,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37289.46 MB 2025-02-15 08:32:32,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 08:32:32,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12392.07 MB 2025-02-15 08:32:32,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26428.98 MB 2025-02-15 08:32:32,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:32:32,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:32:32,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:32:32,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22449.39 MB 2025-02-15 08:32:32,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24338.93 MB 2025-02-15 08:32:32,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:32:32,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 08:32:32,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27728.54 MB 2025-02-15 08:32:32,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:32:32,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25756.35 MB 2025-02-15 08:32:32,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:32:32,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:32:32,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:32:32,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24338.93 MB 2025-02-15 08:32:32,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26580.78 MB 2025-02-15 08:32:32,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:32:32,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27728.54 MB 2025-02-15 08:32:32,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33862.71 MB 2025-02-15 08:32:32,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:32:32,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32125.06 MB 2025-02-15 08:32:32,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:32:32,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:32:32,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:32:32,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22449.39 MB 2025-02-15 08:32:32,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26580.78 MB 2025-02-15 08:32:32,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:32:32,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 08:32:32,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33862.71 MB 2025-02-15 08:32:32,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:32:32,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32125.06 MB 2025-02-15 08:32:32,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:32:32,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:32:32,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:32:32,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28114.32 MB 2025-02-15 08:32:32,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28881.33 MB 2025-02-15 08:32:32,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:32:32,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33862.71 MB 2025-02-15 08:32:32,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 08:32:32,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:32:32,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29589.11 MB 2025-02-15 08:32:32,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:32:32,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:32:32,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:32:32,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29294.21 MB 2025-02-15 08:32:32,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29522.73 MB 2025-02-15 08:32:32,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-15 08:32:32,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 08:32:32,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 08:32:32,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:32,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29753.45 MB 2025-02-15 08:32:32,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:32:32,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:32:32,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.79 seconds 2025-02-15 08:32:32,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17083.40 MB 2025-02-15 08:32:32,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29723.17 MB 2025-02-15 08:32:32,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12639.76 MB 2025-02-15 08:32:32,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43559.94 MB 2025-02-15 08:32:32,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 08:32:32,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9281.99 MB 2025-02-15 08:32:32,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29753.45 MB 2025-02-15 08:32:32,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:32:32,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:32:32,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:32:32,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29723.17 MB 2025-02-15 08:32:32,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22078.31 MB 2025-02-15 08:32:32,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7644.86 MB 2025-02-15 08:32:32,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 08:32:32,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 08:32:32,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:32:32,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32226.85 MB 2025-02-15 08:32:32,876 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 08:32:32,877 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:32:32,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:32:32,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:32:32,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:32:32,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:32:32,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22078.31 MB 2025-02-15 08:32:32,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30490.74 MB 2025-02-15 08:32:32,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-15 08:32:32,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 08:32:32,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42641.39 MB 2025-02-15 08:32:32,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 08:32:32,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30490.74 MB 2025-02-15 08:32:33,043 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 08:32:33,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:33,044 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:32:33,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:33,045 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:32:33,050 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:32:33,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:32:33,051 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:32:33,051 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:33:42,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:42,239 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:33:42,244 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:33:42,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:42,248 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:33:42,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:42,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:33:45,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:33:45,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:33:45,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-15 08:33:45,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-15 08:33:45,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-15 08:33:45,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-15 08:33:45,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51004.83 MB 2025-02-15 08:33:45,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33344.72 MB 2025-02-15 08:33:45,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23722.22 MB 2025-02-15 08:33:45,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:33:45,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:33:45,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:33:45,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-15 08:33:45,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14009.54 MB 2025-02-15 08:33:45,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -892.47 MB 2025-02-15 08:33:45,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15092.39 MB 2025-02-15 08:33:45,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:33:45,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:33:45,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:33:45,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14009.54 MB 2025-02-15 08:33:45,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14025.47 MB 2025-02-15 08:33:45,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15.93 MB 2025-02-15 08:33:45,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14775.43 MB 2025-02-15 08:33:45,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:33:45,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:33:45,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:33:45,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.40 MB 2025-02-15 08:33:45,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.07 MB 2025-02-15 08:33:45,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 56.67 MB 2025-02-15 08:33:45,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14124.60 MB 2025-02-15 08:33:45,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:33:45,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:33:45,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:33:45,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.07 MB 2025-02-15 08:33:45,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14149.39 MB 2025-02-15 08:33:45,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 67.32 MB 2025-02-15 08:33:45,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14315.66 MB 2025-02-15 08:33:45,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:33:45,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:33:45,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:33:45,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.40 MB 2025-02-15 08:33:45,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14149.39 MB 2025-02-15 08:33:45,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.99 MB 2025-02-15 08:33:45,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17660.12 MB 2025-02-15 08:33:45,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14315.66 MB 2025-02-15 08:33:45,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:33:45,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:33:45,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:33:45,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.40 MB 2025-02-15 08:33:45,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14218.41 MB 2025-02-15 08:33:45,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 23.01 MB 2025-02-15 08:33:45,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17660.12 MB 2025-02-15 08:33:45,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17668.51 MB 2025-02-15 08:33:45,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8.39 MB 2025-02-15 08:33:45,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14247.16 MB 2025-02-15 08:33:45,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:33:45,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:33:45,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:33:45,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14230.81 MB 2025-02-15 08:33:45,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14247.54 MB 2025-02-15 08:33:45,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16.73 MB 2025-02-15 08:33:45,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17668.51 MB 2025-02-15 08:33:45,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17668.51 MB 2025-02-15 08:33:45,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14247.54 MB 2025-02-15 08:33:45,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:33:45,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:33:45,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.99 seconds 2025-02-15 08:33:45,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-15 08:33:45,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14278.15 MB 2025-02-15 08:33:45,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 668.38 MB 2025-02-15 08:33:45,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51004.83 MB 2025-02-15 08:33:45,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17668.51 MB 2025-02-15 08:33:45,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33336.33 MB 2025-02-15 08:33:45,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14278.15 MB 2025-02-15 08:33:45,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:33:45,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:33:45,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 08:33:45,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14278.15 MB 2025-02-15 08:33:45,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14737.11 MB 2025-02-15 08:33:45,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.96 MB 2025-02-15 08:33:45,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17668.51 MB 2025-02-15 08:33:45,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17670.60 MB 2025-02-15 08:33:45,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 08:33:45,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14783.01 MB 2025-02-15 08:33:45,302 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1231, cut from 1233 2025-02-15 08:33:45,303 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:33:45,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:33:45,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:33:45,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:33:45,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:33:45,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14142.10 MB 2025-02-15 08:33:45,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15426.66 MB 2025-02-15 08:33:45,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1284.57 MB 2025-02-15 08:33:45,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17670.60 MB 2025-02-15 08:33:45,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17670.60 MB 2025-02-15 08:33:45,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:33:45,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15426.66 MB 2025-02-15 08:33:45,329 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1023] 2025-02-15 08:33:45,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:45,331 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:33:45,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:45,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:33:45,337 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:33:45,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:33:45,338 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:33:45,338 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:34:37,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:34:37,026 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:34:37,031 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:34:37,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:34:37,035 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1589, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:34:37,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:34:37,037 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1589, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:35:01,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:35:01,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:35:01,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.47 seconds 2025-02-15 08:35:01,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:01,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24042.29 MB 2025-02-15 08:35:01,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29665.67 MB 2025-02-15 08:35:01,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.38 MB 2025-02-15 08:35:01,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29536.29 MB 2025-02-15 08:35:01,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33181.14 MB 2025-02-15 08:35:01,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3644.85 MB 2025-02-15 08:35:01,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38496.50 MB 2025-02-15 08:35:01,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:35:01,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:35:01,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 08:35:01,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:01,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29665.67 MB 2025-02-15 08:35:01,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24040.20 MB 2025-02-15 08:35:01,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5625.47 MB 2025-02-15 08:35:01,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33181.14 MB 2025-02-15 08:35:01,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46017.81 MB 2025-02-15 08:35:01,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12836.67 MB 2025-02-15 08:35:01,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45774.34 MB 2025-02-15 08:35:03,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:35:03,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:35:03,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 08:35:03,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:03,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24040.20 MB 2025-02-15 08:35:03,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24571.05 MB 2025-02-15 08:35:03,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:35:03,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46017.81 MB 2025-02-15 08:35:03,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26868.71 MB 2025-02-15 08:35:03,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19149.09 MB 2025-02-15 08:35:03,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28549.59 MB 2025-02-15 08:35:03,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:35:03,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:35:03,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:35:03,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:03,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24571.05 MB 2025-02-15 08:35:03,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26460.27 MB 2025-02-15 08:35:03,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-15 08:35:03,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26868.71 MB 2025-02-15 08:35:03,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29699.87 MB 2025-02-15 08:35:03,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:35:03,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.70 MB 2025-02-15 08:35:03,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:35:03,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:35:03,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:35:03,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:03,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26460.27 MB 2025-02-15 08:35:03,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28702.12 MB 2025-02-15 08:35:03,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:35:03,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29699.87 MB 2025-02-15 08:35:03,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35834.04 MB 2025-02-15 08:35:03,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:35:03,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34246.41 MB 2025-02-15 08:35:03,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:35:03,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:35:03,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:35:03,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:03,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24571.05 MB 2025-02-15 08:35:03,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28702.12 MB 2025-02-15 08:35:03,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.08 MB 2025-02-15 08:35:03,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26868.71 MB 2025-02-15 08:35:03,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35834.04 MB 2025-02-15 08:35:03,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:35:03,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34246.41 MB 2025-02-15 08:35:03,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:35:03,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:35:03,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 08:35:03,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:03,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30235.67 MB 2025-02-15 08:35:03,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31002.67 MB 2025-02-15 08:35:03,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:35:03,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35834.04 MB 2025-02-15 08:35:03,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36249.27 MB 2025-02-15 08:35:03,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:35:03,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31710.46 MB 2025-02-15 08:35:04,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:35:04,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:35:04,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:35:04,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:04,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31415.56 MB 2025-02-15 08:35:04,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31643.42 MB 2025-02-15 08:35:04,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.86 MB 2025-02-15 08:35:04,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36249.27 MB 2025-02-15 08:35:04,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36249.27 MB 2025-02-15 08:35:04,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:35:04,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.58 MB 2025-02-15 08:35:04,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:35:04,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:35:04,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.98 seconds 2025-02-15 08:35:04,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:04,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18505.50 MB 2025-02-15 08:35:04,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31843.51 MB 2025-02-15 08:35:04,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13338.01 MB 2025-02-15 08:35:04,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23997.71 MB 2025-02-15 08:35:04,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36249.27 MB 2025-02-15 08:35:04,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12251.56 MB 2025-02-15 08:35:04,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.58 MB 2025-02-15 08:35:04,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:35:04,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:35:04,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:35:04,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:04,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31843.51 MB 2025-02-15 08:35:04,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23495.10 MB 2025-02-15 08:35:04,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8348.41 MB 2025-02-15 08:35:04,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36249.27 MB 2025-02-15 08:35:04,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36249.27 MB 2025-02-15 08:35:04,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:35:04,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34342.89 MB 2025-02-15 08:35:04,304 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 08:35:04,305 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:35:04,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:35:04,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:35:04,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:35:04,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:04,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23495.10 MB 2025-02-15 08:35:04,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.50 MB 2025-02-15 08:35:04,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 08:35:04,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36249.27 MB 2025-02-15 08:35:04,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44600.13 MB 2025-02-15 08:35:04,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 08:35:04,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31892.50 MB 2025-02-15 08:35:04,468 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 08:35:04,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:04,469 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:35:04,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:04,470 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:35:04,475 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:35:04,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:04,476 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:35:04,476 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:35:14,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:14,086 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:35:14,091 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:35:14,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:14,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:35:14,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:14,095 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:35:32,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:35:32,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:35:32,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.47 seconds 2025-02-15 08:35:32,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:32,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.94 MB 2025-02-15 08:35:32,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25430.13 MB 2025-02-15 08:35:32,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-15 08:35:32,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52950.99 MB 2025-02-15 08:35:32,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27579.65 MB 2025-02-15 08:35:32,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25371.34 MB 2025-02-15 08:35:32,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34329.00 MB 2025-02-15 08:35:32,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:35:32,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:35:32,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 08:35:32,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:32,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25430.13 MB 2025-02-15 08:35:32,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21944.54 MB 2025-02-15 08:35:32,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3485.59 MB 2025-02-15 08:35:32,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27579.65 MB 2025-02-15 08:35:32,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-15 08:35:32,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10448.01 MB 2025-02-15 08:35:32,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37979.56 MB 2025-02-15 08:35:34,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:35:34,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:35:34,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:35:34,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:34,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21944.54 MB 2025-02-15 08:35:34,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22475.38 MB 2025-02-15 08:35:34,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:35:34,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-15 08:35:34,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25505.56 MB 2025-02-15 08:35:34,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12522.09 MB 2025-02-15 08:35:34,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26456.89 MB 2025-02-15 08:35:34,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:35:34,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:35:34,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:35:34,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:34,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22475.38 MB 2025-02-15 08:35:34,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24364.92 MB 2025-02-15 08:35:34,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:35:34,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25505.56 MB 2025-02-15 08:35:34,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27393.00 MB 2025-02-15 08:35:34,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:35:34,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25782.35 MB 2025-02-15 08:35:34,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:35:34,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:35:34,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 08:35:34,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:34,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24364.92 MB 2025-02-15 08:35:34,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26606.77 MB 2025-02-15 08:35:34,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:35:34,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27393.00 MB 2025-02-15 08:35:34,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33999.03 MB 2025-02-15 08:35:34,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:35:34,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32151.06 MB 2025-02-15 08:35:34,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:35:34,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:35:34,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:35:34,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:34,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22475.38 MB 2025-02-15 08:35:34,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26606.77 MB 2025-02-15 08:35:34,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:35:34,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25505.56 MB 2025-02-15 08:35:34,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33999.03 MB 2025-02-15 08:35:34,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 08:35:34,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32151.06 MB 2025-02-15 08:35:35,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:35:35,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:35:35,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:35:35,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:35,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28140.32 MB 2025-02-15 08:35:35,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28907.32 MB 2025-02-15 08:35:35,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:35:35,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33999.03 MB 2025-02-15 08:35:35,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-15 08:35:35,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:35:35,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29615.11 MB 2025-02-15 08:35:35,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:35:35,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:35:35,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:35:35,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:35,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29320.21 MB 2025-02-15 08:35:35,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29545.90 MB 2025-02-15 08:35:35,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.69 MB 2025-02-15 08:35:35,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34414.26 MB 2025-02-15 08:35:35,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-15 08:35:35,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:35:35,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.73 MB 2025-02-15 08:35:35,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:35:35,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:35:35,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.93 seconds 2025-02-15 08:35:35,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:35,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17100.83 MB 2025-02-15 08:35:35,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29746.75 MB 2025-02-15 08:35:35,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12645.93 MB 2025-02-15 08:35:35,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52950.99 MB 2025-02-15 08:35:35,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-15 08:35:35,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18536.73 MB 2025-02-15 08:35:35,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.73 MB 2025-02-15 08:35:35,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:35:35,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:35:35,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:35:35,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:35,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29746.75 MB 2025-02-15 08:35:35,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22088.24 MB 2025-02-15 08:35:35,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7658.51 MB 2025-02-15 08:35:35,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34414.26 MB 2025-02-15 08:35:35,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-15 08:35:35,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:35:35,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32243.98 MB 2025-02-15 08:35:35,312 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 08:35:35,313 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:35:35,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:35:35,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:35:35,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:35:35,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:35:35,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22088.24 MB 2025-02-15 08:35:35,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30479.29 MB 2025-02-15 08:35:35,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 08:35:35,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34414.26 MB 2025-02-15 08:35:35,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42756.73 MB 2025-02-15 08:35:35,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 08:35:35,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30479.29 MB 2025-02-15 08:35:35,476 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 08:35:35,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:35,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:35:35,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:35,478 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:35:35,483 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:35:35,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:35:35,484 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:35:35,484 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:36:34,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:34,276 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:36:34,281 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:36:34,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:34,284 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:36:34,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:34,285 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:36:37,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:36:37,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:36:37,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.77 seconds 2025-02-15 08:36:37,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:37,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-15 08:36:37,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-15 08:36:37,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-15 08:36:37,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51099.21 MB 2025-02-15 08:36:37,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19440.60 MB 2025-02-15 08:36:37,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31658.61 MB 2025-02-15 08:36:37,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-15 08:36:37,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:36:37,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:36:37,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:36:37,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:37,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-15 08:36:37,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15142.35 MB 2025-02-15 08:36:37,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.87 MB 2025-02-15 08:36:37,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19440.60 MB 2025-02-15 08:36:37,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19440.60 MB 2025-02-15 08:36:37,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:36:37,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17335.69 MB 2025-02-15 08:36:37,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:36:37,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:36:37,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-15 08:36:37,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:37,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15142.35 MB 2025-02-15 08:36:37,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.25 MB 2025-02-15 08:36:37,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 08:36:37,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19440.60 MB 2025-02-15 08:36:37,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17970.50 MB 2025-02-15 08:36:37,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1470.10 MB 2025-02-15 08:36:37,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.04 MB 2025-02-15 08:36:37,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:36:37,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:36:37,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:36:37,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:37,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-15 08:36:37,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.10 MB 2025-02-15 08:36:37,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 08:36:37,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17970.50 MB 2025-02-15 08:36:37,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18389.93 MB 2025-02-15 08:36:37,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 08:36:37,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.31 MB 2025-02-15 08:36:38,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:36:38,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:36:38,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:36:38,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.10 MB 2025-02-15 08:36:38,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-15 08:36:38,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 08:36:38,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18389.93 MB 2025-02-15 08:36:38,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21116.22 MB 2025-02-15 08:36:38,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2726.30 MB 2025-02-15 08:36:38,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.38 MB 2025-02-15 08:36:38,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:36:38,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:36:38,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:36:38,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-15 08:36:38,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-15 08:36:38,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 08:36:38,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17970.50 MB 2025-02-15 08:36:38,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21116.22 MB 2025-02-15 08:36:38,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3145.73 MB 2025-02-15 08:36:38,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.38 MB 2025-02-15 08:36:38,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:36:38,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:36:38,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:36:38,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.75 MB 2025-02-15 08:36:38,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.06 MB 2025-02-15 08:36:38,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 08:36:38,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21116.22 MB 2025-02-15 08:36:38,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21296.58 MB 2025-02-15 08:36:38,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 08:36:38,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18544.54 MB 2025-02-15 08:36:38,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:36:38,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:36:38,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:36:38,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18406.77 MB 2025-02-15 08:36:38,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18633.76 MB 2025-02-15 08:36:38,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.99 MB 2025-02-15 08:36:38,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21296.58 MB 2025-02-15 08:36:38,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21296.58 MB 2025-02-15 08:36:38,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:36:38,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18650.95 MB 2025-02-15 08:36:38,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:36:38,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:36:38,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.83 seconds 2025-02-15 08:36:38,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-15 08:36:38,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.54 MB 2025-02-15 08:36:38,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.18 MB 2025-02-15 08:36:38,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51099.21 MB 2025-02-15 08:36:38,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21296.58 MB 2025-02-15 08:36:38,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29802.63 MB 2025-02-15 08:36:38,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18834.54 MB 2025-02-15 08:36:38,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:36:38,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:36:38,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:36:38,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18834.54 MB 2025-02-15 08:36:38,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17540.43 MB 2025-02-15 08:36:38,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1294.11 MB 2025-02-15 08:36:38,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21296.58 MB 2025-02-15 08:36:38,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21296.58 MB 2025-02-15 08:36:38,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:36:38,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19069.30 MB 2025-02-15 08:36:38,407 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 08:36:38,407 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:36:38,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:36:38,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:36:38,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:36:38,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:36:38,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17540.43 MB 2025-02-15 08:36:38,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25966.93 MB 2025-02-15 08:36:38,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 08:36:38,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21296.58 MB 2025-02-15 08:36:38,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31769.76 MB 2025-02-15 08:36:38,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 08:36:38,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25966.93 MB 2025-02-15 08:36:38,573 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 08:36:38,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:38,575 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:36:38,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:38,576 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:36:38,582 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:36:38,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:38,583 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:36:38,583 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:36:52,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:52,165 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:36:52,170 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:36:52,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:52,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:36:52,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:36:52,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:37:11,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:37:11,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:37:11,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.59 seconds 2025-02-15 08:37:11,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:11,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-15 08:37:11,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-15 08:37:11,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-15 08:37:11,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44335.89 MB 2025-02-15 08:37:11,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37721.47 MB 2025-02-15 08:37:11,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6614.42 MB 2025-02-15 08:37:11,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-15 08:37:11,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:37:11,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:37:11,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:37:11,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:11,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-15 08:37:11,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-15 08:37:11,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-15 08:37:11,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37721.47 MB 2025-02-15 08:37:11,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46609.20 MB 2025-02-15 08:37:11,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8887.73 MB 2025-02-15 08:37:11,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39643.54 MB 2025-02-15 08:37:13,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:37:13,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:37:13,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:37:13,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:13,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-15 08:37:13,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-15 08:37:13,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:37:13,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46609.20 MB 2025-02-15 08:37:13,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29047.65 MB 2025-02-15 08:37:13,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17561.55 MB 2025-02-15 08:37:13,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26873.98 MB 2025-02-15 08:37:13,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:37:13,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:37:13,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:37:13,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:13,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 08:37:13,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-15 08:37:13,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:37:13,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29047.65 MB 2025-02-15 08:37:13,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29047.65 MB 2025-02-15 08:37:13,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:37:13,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-15 08:37:13,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:37:13,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:37:13,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:37:13,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:13,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-15 08:37:13,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 08:37:13,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:37:13,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29047.65 MB 2025-02-15 08:37:13,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34709.96 MB 2025-02-15 08:37:13,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:37:13,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 08:37:13,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:37:13,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:37:13,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:37:13,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:13,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 08:37:13,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 08:37:13,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:37:13,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29047.65 MB 2025-02-15 08:37:13,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34709.96 MB 2025-02-15 08:37:13,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:37:13,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 08:37:14,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:37:14,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:37:14,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:37:14,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:14,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-15 08:37:14,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-15 08:37:14,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:37:14,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34709.96 MB 2025-02-15 08:37:14,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-15 08:37:14,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 08:37:14,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-15 08:37:14,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:37:14,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:37:14,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:37:14,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:14,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-15 08:37:14,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29967.46 MB 2025-02-15 08:37:14,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.20 MB 2025-02-15 08:37:14,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-15 08:37:14,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-15 08:37:14,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:37:14,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30207.39 MB 2025-02-15 08:37:14,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:37:14,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:37:14,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.00 seconds 2025-02-15 08:37:14,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:14,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-15 08:37:14,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30168.46 MB 2025-02-15 08:37:14,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12785.42 MB 2025-02-15 08:37:14,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44335.89 MB 2025-02-15 08:37:14,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-15 08:37:14,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9212.79 MB 2025-02-15 08:37:14,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30207.39 MB 2025-02-15 08:37:14,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:37:14,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:37:14,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:37:14,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:14,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30168.46 MB 2025-02-15 08:37:14,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22386.28 MB 2025-02-15 08:37:14,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7782.17 MB 2025-02-15 08:37:14,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-15 08:37:14,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-15 08:37:14,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:37:14,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32679.20 MB 2025-02-15 08:37:14,468 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 08:37:14,468 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:37:14,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:37:14,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:37:14,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:37:14,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:37:14,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22386.28 MB 2025-02-15 08:37:14,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30821.88 MB 2025-02-15 08:37:14,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 08:37:14,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-15 08:37:14,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43511.71 MB 2025-02-15 08:37:14,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 08:37:14,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30821.88 MB 2025-02-15 08:37:14,633 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 08:37:14,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:37:14,634 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:37:14,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:37:14,635 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:37:14,640 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:37:14,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:37:14,641 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:37:14,641 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:38:08,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:08,077 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:38:08,085 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:38:08,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:08,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:38:08,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:08,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:38:11,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:38:11,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:38:11,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.59 seconds 2025-02-15 08:38:11,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:11,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14536.54 MB 2025-02-15 08:38:11,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.80 MB 2025-02-15 08:38:11,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 796.26 MB 2025-02-15 08:38:11,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51900.32 MB 2025-02-15 08:38:11,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 08:38:11,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33533.46 MB 2025-02-15 08:38:11,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24235.32 MB 2025-02-15 08:38:11,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:38:11,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:38:11,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:38:11,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:11,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.80 MB 2025-02-15 08:38:11,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.65 MB 2025-02-15 08:38:11,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -21.16 MB 2025-02-15 08:38:11,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 08:38:11,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18958.25 MB 2025-02-15 08:38:11,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 591.40 MB 2025-02-15 08:38:11,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17678.95 MB 2025-02-15 08:38:12,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:38:12,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:38:12,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 08:38:12,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.65 MB 2025-02-15 08:38:12,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15533.28 MB 2025-02-15 08:38:12,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-15 08:38:12,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18958.25 MB 2025-02-15 08:38:12,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18889.05 MB 2025-02-15 08:38:12,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -69.21 MB 2025-02-15 08:38:12,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19482.34 MB 2025-02-15 08:38:12,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:38:12,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:38:12,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:38:12,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15533.21 MB 2025-02-15 08:38:12,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16321.90 MB 2025-02-15 08:38:12,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-15 08:38:12,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 08:38:12,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18889.05 MB 2025-02-15 08:38:12,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:38:12,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16913.68 MB 2025-02-15 08:38:12,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:38:12,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:38:12,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:38:12,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16321.90 MB 2025-02-15 08:38:12,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17257.91 MB 2025-02-15 08:38:12,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-15 08:38:12,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 08:38:12,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20860.37 MB 2025-02-15 08:38:12,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1971.32 MB 2025-02-15 08:38:12,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19573.92 MB 2025-02-15 08:38:12,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:38:12,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:38:12,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:38:12,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15533.21 MB 2025-02-15 08:38:12,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17257.91 MB 2025-02-15 08:38:12,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-15 08:38:12,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18889.05 MB 2025-02-15 08:38:12,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20860.37 MB 2025-02-15 08:38:12,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1971.32 MB 2025-02-15 08:38:12,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19573.92 MB 2025-02-15 08:38:12,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:38:12,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:38:12,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:38:12,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17898.17 MB 2025-02-15 08:38:12,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18218.65 MB 2025-02-15 08:38:12,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.49 MB 2025-02-15 08:38:12,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20860.37 MB 2025-02-15 08:38:12,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21030.24 MB 2025-02-15 08:38:12,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 08:38:12,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18524.18 MB 2025-02-15 08:38:12,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:38:12,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:38:12,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:38:12,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18391.04 MB 2025-02-15 08:38:12,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18616.85 MB 2025-02-15 08:38:12,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.81 MB 2025-02-15 08:38:12,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21030.24 MB 2025-02-15 08:38:12,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21030.24 MB 2025-02-15 08:38:12,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:38:12,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18643.19 MB 2025-02-15 08:38:12,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:38:12,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:38:12,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.62 seconds 2025-02-15 08:38:12,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13752.62 MB 2025-02-15 08:38:12,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18817.70 MB 2025-02-15 08:38:12,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5065.07 MB 2025-02-15 08:38:12,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51900.32 MB 2025-02-15 08:38:12,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21030.24 MB 2025-02-15 08:38:12,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30870.08 MB 2025-02-15 08:38:12,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18817.70 MB 2025-02-15 08:38:12,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:38:12,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:38:12,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:38:12,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:12,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18817.70 MB 2025-02-15 08:38:12,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17654.25 MB 2025-02-15 08:38:12,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1163.45 MB 2025-02-15 08:38:12,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21030.24 MB 2025-02-15 08:38:12,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21030.24 MB 2025-02-15 08:38:12,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:38:12,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19419.83 MB 2025-02-15 08:38:13,004 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 08:38:13,005 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:38:13,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:38:13,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:38:13,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:38:13,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:38:13,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17654.25 MB 2025-02-15 08:38:13,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26084.65 MB 2025-02-15 08:38:13,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 08:38:13,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21030.24 MB 2025-02-15 08:38:13,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29410.46 MB 2025-02-15 08:38:13,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 08:38:13,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26084.65 MB 2025-02-15 08:38:13,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 08:38:13,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:13,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:38:13,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:13,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:38:13,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:38:13,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:38:13,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:38:13,181 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:39:08,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:08,603 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:39:08,609 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:39:08,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:08,614 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:39:08,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:08,615 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:39:26,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:39:26,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:39:26,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.46 seconds 2025-02-15 08:39:26,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:26,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20843.33 MB 2025-02-15 08:39:26,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24842.60 MB 2025-02-15 08:39:26,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3999.27 MB 2025-02-15 08:39:26,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37790.68 MB 2025-02-15 08:39:26,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28856.81 MB 2025-02-15 08:39:26,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8933.87 MB 2025-02-15 08:39:26,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33712.09 MB 2025-02-15 08:39:26,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:39:26,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:39:26,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:39:26,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:26,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24842.60 MB 2025-02-15 08:39:26,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21654.02 MB 2025-02-15 08:39:26,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3188.58 MB 2025-02-15 08:39:26,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28856.81 MB 2025-02-15 08:39:26,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38872.81 MB 2025-02-15 08:39:26,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10016.00 MB 2025-02-15 08:39:26,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36956.27 MB 2025-02-15 08:39:28,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:39:28,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:39:28,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 08:39:28,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21654.02 MB 2025-02-15 08:39:28,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22184.86 MB 2025-02-15 08:39:28,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:39:28,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38872.81 MB 2025-02-15 08:39:28,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26981.96 MB 2025-02-15 08:39:28,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11890.85 MB 2025-02-15 08:39:28,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26163.41 MB 2025-02-15 08:39:28,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:39:28,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:39:28,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:39:28,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22184.86 MB 2025-02-15 08:39:28,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24074.39 MB 2025-02-15 08:39:28,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:39:28,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26981.96 MB 2025-02-15 08:39:28,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27925.68 MB 2025-02-15 08:39:28,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 08:39:28,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25491.82 MB 2025-02-15 08:39:28,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:39:28,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:39:28,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:39:28,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24074.39 MB 2025-02-15 08:39:28,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26316.25 MB 2025-02-15 08:39:28,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:39:28,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27925.68 MB 2025-02-15 08:39:28,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33587.99 MB 2025-02-15 08:39:28,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:39:28,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31860.53 MB 2025-02-15 08:39:28,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:39:28,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:39:28,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:39:28,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22184.86 MB 2025-02-15 08:39:28,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26316.25 MB 2025-02-15 08:39:28,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:39:28,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26981.96 MB 2025-02-15 08:39:28,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33587.99 MB 2025-02-15 08:39:28,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:39:28,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31860.53 MB 2025-02-15 08:39:28,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:39:28,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:39:28,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:39:28,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27849.79 MB 2025-02-15 08:39:28,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.79 MB 2025-02-15 08:39:28,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:39:28,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33587.99 MB 2025-02-15 08:39:28,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 08:39:28,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:39:28,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29324.58 MB 2025-02-15 08:39:28,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:39:28,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:39:28,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:39:28,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29029.68 MB 2025-02-15 08:39:28,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29257.28 MB 2025-02-15 08:39:28,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.59 MB 2025-02-15 08:39:28,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 08:39:28,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 08:39:28,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:39:28,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29490.45 MB 2025-02-15 08:39:28,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:39:28,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:39:28,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.89 seconds 2025-02-15 08:39:28,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16906.32 MB 2025-02-15 08:39:28,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29458.20 MB 2025-02-15 08:39:28,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12551.88 MB 2025-02-15 08:39:28,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37790.68 MB 2025-02-15 08:39:28,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 08:39:28,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3785.36 MB 2025-02-15 08:39:28,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29490.45 MB 2025-02-15 08:39:28,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:39:28,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:39:28,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:39:28,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29458.20 MB 2025-02-15 08:39:28,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21908.16 MB 2025-02-15 08:39:28,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7550.04 MB 2025-02-15 08:39:28,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 08:39:28,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 08:39:28,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:39:28,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31968.02 MB 2025-02-15 08:39:28,796 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 08:39:28,796 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:39:28,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:39:28,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:39:28,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:39:28,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:39:28,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.16 MB 2025-02-15 08:39:28,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.46 MB 2025-02-15 08:39:28,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 08:39:28,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 08:39:28,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42389.73 MB 2025-02-15 08:39:28,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 08:39:28,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30341.46 MB 2025-02-15 08:39:28,960 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 08:39:28,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:28,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:39:28,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:28,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:39:28,967 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:39:28,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:39:28,969 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:39:28,969 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:40:13,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:13,900 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:40:13,905 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:40:13,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:13,909 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1260, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:40:13,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:13,910 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1260, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:40:33,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:40:33,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:40:33,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.53 seconds 2025-02-15 08:40:33,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:33,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21748.59 MB 2025-02-15 08:40:33,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26207.66 MB 2025-02-15 08:40:33,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4459.07 MB 2025-02-15 08:40:33,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50774.15 MB 2025-02-15 08:40:33,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37702.60 MB 2025-02-15 08:40:33,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13071.55 MB 2025-02-15 08:40:33,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35070.33 MB 2025-02-15 08:40:33,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:40:33,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:40:33,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:40:33,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:33,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26207.66 MB 2025-02-15 08:40:33,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22328.20 MB 2025-02-15 08:40:33,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3879.46 MB 2025-02-15 08:40:33,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37702.60 MB 2025-02-15 08:40:33,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46556.77 MB 2025-02-15 08:40:33,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8854.18 MB 2025-02-15 08:40:33,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39515.78 MB 2025-02-15 08:40:35,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:40:35,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:40:35,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:40:35,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22328.20 MB 2025-02-15 08:40:35,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22859.04 MB 2025-02-15 08:40:35,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:40:35,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46556.77 MB 2025-02-15 08:40:35,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33241.96 MB 2025-02-15 08:40:35,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13314.82 MB 2025-02-15 08:40:35,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26837.59 MB 2025-02-15 08:40:35,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:40:35,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:40:35,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:40:35,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-15 08:40:35,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24748.57 MB 2025-02-15 08:40:35,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:40:35,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33241.96 MB 2025-02-15 08:40:35,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33241.96 MB 2025-02-15 08:40:35,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:40:35,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.00 MB 2025-02-15 08:40:35,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:40:35,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:40:35,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:40:35,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24748.57 MB 2025-02-15 08:40:35,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-15 08:40:35,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:40:35,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33241.96 MB 2025-02-15 08:40:35,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33713.82 MB 2025-02-15 08:40:35,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 08:40:35,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-15 08:40:35,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:40:35,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:40:35,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:40:35,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-15 08:40:35,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-15 08:40:35,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:40:35,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33241.96 MB 2025-02-15 08:40:35,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33713.82 MB 2025-02-15 08:40:35,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 08:40:35,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-15 08:40:35,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:40:35,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:40:35,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:40:35,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28523.97 MB 2025-02-15 08:40:35,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.97 MB 2025-02-15 08:40:35,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:40:35,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33713.82 MB 2025-02-15 08:40:35,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34131.15 MB 2025-02-15 08:40:35,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:40:35,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29998.76 MB 2025-02-15 08:40:35,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:40:35,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:40:35,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:40:35,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29703.86 MB 2025-02-15 08:40:35,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29932.04 MB 2025-02-15 08:40:35,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 08:40:35,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34131.15 MB 2025-02-15 08:40:35,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34131.15 MB 2025-02-15 08:40:35,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:40:35,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30164.65 MB 2025-02-15 08:40:35,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:40:35,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:40:35,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.96 seconds 2025-02-15 08:40:35,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:35,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17358.65 MB 2025-02-15 08:40:35,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30132.13 MB 2025-02-15 08:40:35,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12773.48 MB 2025-02-15 08:40:35,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50774.15 MB 2025-02-15 08:40:35,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34131.15 MB 2025-02-15 08:40:35,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16643.00 MB 2025-02-15 08:40:35,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30164.65 MB 2025-02-15 08:40:36,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:40:36,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:40:36,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:40:36,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:36,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30132.13 MB 2025-02-15 08:40:36,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22348.56 MB 2025-02-15 08:40:36,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7783.57 MB 2025-02-15 08:40:36,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34131.15 MB 2025-02-15 08:40:36,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34131.15 MB 2025-02-15 08:40:36,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:40:36,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32631.51 MB 2025-02-15 08:40:36,159 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 08:40:36,159 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:40:36,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:40:36,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:40:36,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:40:36,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:40:36,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.56 MB 2025-02-15 08:40:36,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30745.96 MB 2025-02-15 08:40:36,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 08:40:36,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34131.15 MB 2025-02-15 08:40:36,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38306.58 MB 2025-02-15 08:40:36,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 08:40:36,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30745.96 MB 2025-02-15 08:40:36,323 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 08:40:36,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:36,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:40:36,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:36,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:40:36,330 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:40:36,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:40:36,331 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:40:36,332 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:42:12,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:12,604 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:42:12,609 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:42:12,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:12,613 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1001, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:42:12,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:12,614 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1001, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:42:27,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:42:27,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:42:27,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.35 seconds 2025-02-15 08:42:27,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:27,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19943.83 MB 2025-02-15 08:42:27,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23486.32 MB 2025-02-15 08:42:27,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3542.48 MB 2025-02-15 08:42:27,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46657.44 MB 2025-02-15 08:42:27,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28408.02 MB 2025-02-15 08:42:27,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18249.42 MB 2025-02-15 08:42:27,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32359.61 MB 2025-02-15 08:42:28,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:42:28,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:42:28,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:42:28,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:28,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23486.32 MB 2025-02-15 08:42:28,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20982.79 MB 2025-02-15 08:42:28,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2503.53 MB 2025-02-15 08:42:28,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28408.02 MB 2025-02-15 08:42:28,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37016.83 MB 2025-02-15 08:42:28,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8608.81 MB 2025-02-15 08:42:28,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34196.55 MB 2025-02-15 08:42:29,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:42:29,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:42:29,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 08:42:29,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:29,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.79 MB 2025-02-15 08:42:29,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21513.63 MB 2025-02-15 08:42:29,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:42:29,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37016.83 MB 2025-02-15 08:42:29,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26988.25 MB 2025-02-15 08:42:29,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10028.58 MB 2025-02-15 08:42:29,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25492.17 MB 2025-02-15 08:42:29,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:42:29,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:42:29,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:42:29,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:29,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21513.63 MB 2025-02-15 08:42:29,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23403.16 MB 2025-02-15 08:42:29,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:42:29,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26988.25 MB 2025-02-15 08:42:29,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27931.97 MB 2025-02-15 08:42:29,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 08:42:29,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24820.59 MB 2025-02-15 08:42:30,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:42:30,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:42:30,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:42:30,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23403.16 MB 2025-02-15 08:42:30,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25645.02 MB 2025-02-15 08:42:30,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:42:30,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27931.97 MB 2025-02-15 08:42:30,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33594.28 MB 2025-02-15 08:42:30,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:42:30,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31189.30 MB 2025-02-15 08:42:30,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:42:30,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:42:30,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:42:30,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21513.63 MB 2025-02-15 08:42:30,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25645.02 MB 2025-02-15 08:42:30,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:42:30,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26988.25 MB 2025-02-15 08:42:30,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33594.28 MB 2025-02-15 08:42:30,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:42:30,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31189.30 MB 2025-02-15 08:42:30,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:42:30,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:42:30,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:42:30,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27178.56 MB 2025-02-15 08:42:30,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27945.56 MB 2025-02-15 08:42:30,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:42:30,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33594.28 MB 2025-02-15 08:42:30,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34007.42 MB 2025-02-15 08:42:30,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 08:42:30,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28653.35 MB 2025-02-15 08:42:30,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:42:30,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:42:30,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:42:30,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28358.45 MB 2025-02-15 08:42:30,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28585.33 MB 2025-02-15 08:42:30,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.88 MB 2025-02-15 08:42:30,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34007.42 MB 2025-02-15 08:42:30,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34007.42 MB 2025-02-15 08:42:30,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:42:30,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28793.58 MB 2025-02-15 08:42:30,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:42:30,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:42:30,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.75 seconds 2025-02-15 08:42:30,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16456.27 MB 2025-02-15 08:42:30,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28786.19 MB 2025-02-15 08:42:30,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12329.92 MB 2025-02-15 08:42:30,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46657.44 MB 2025-02-15 08:42:30,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34007.42 MB 2025-02-15 08:42:30,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12650.02 MB 2025-02-15 08:42:30,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28793.58 MB 2025-02-15 08:42:30,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:42:30,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:42:30,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:42:30,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28786.19 MB 2025-02-15 08:42:30,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21445.12 MB 2025-02-15 08:42:30,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.07 MB 2025-02-15 08:42:30,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34007.42 MB 2025-02-15 08:42:30,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34007.42 MB 2025-02-15 08:42:30,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:42:30,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31284.64 MB 2025-02-15 08:42:30,654 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 08:42:30,654 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:42:30,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:42:30,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:42:30,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:42:30,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:42:30,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21445.12 MB 2025-02-15 08:42:30,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29840.33 MB 2025-02-15 08:42:30,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 08:42:30,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34007.42 MB 2025-02-15 08:42:30,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44440.75 MB 2025-02-15 08:42:30,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10433.33 MB 2025-02-15 08:42:30,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29840.33 MB 2025-02-15 08:42:30,819 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 08:42:30,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:30,820 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:42:30,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:30,821 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:42:30,826 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:42:30,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:42:30,827 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:42:30,827 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:43:05,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:05,905 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:43:05,910 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:43:05,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:05,914 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1912, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:43:05,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:05,915 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1912, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:43:35,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:43:35,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:43:35,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.67 seconds 2025-02-15 08:43:35,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:35,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26291.83 MB 2025-02-15 08:43:35,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33059.34 MB 2025-02-15 08:43:35,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6767.51 MB 2025-02-15 08:43:35,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52787.41 MB 2025-02-15 08:43:35,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39942.36 MB 2025-02-15 08:43:35,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12845.06 MB 2025-02-15 08:43:35,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41878.49 MB 2025-02-15 08:43:35,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:43:35,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:43:35,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 08:43:35,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:35,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33059.34 MB 2025-02-15 08:43:35,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.74 MB 2025-02-15 08:43:35,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.60 MB 2025-02-15 08:43:35,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-15 08:43:35,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53208.94 MB 2025-02-15 08:43:35,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13266.58 MB 2025-02-15 08:43:35,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50879.03 MB 2025-02-15 08:43:37,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:43:37,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:43:37,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:43:37,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:37,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25717.74 MB 2025-02-15 08:43:37,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26248.58 MB 2025-02-15 08:43:37,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:43:37,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53208.94 MB 2025-02-15 08:43:37,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34590.43 MB 2025-02-15 08:43:37,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18618.52 MB 2025-02-15 08:43:37,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30227.13 MB 2025-02-15 08:43:37,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:43:37,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:43:37,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:43:37,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:37,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26248.58 MB 2025-02-15 08:43:37,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28138.12 MB 2025-02-15 08:43:37,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:43:37,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34590.43 MB 2025-02-15 08:43:37,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34590.43 MB 2025-02-15 08:43:37,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:43:37,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29555.55 MB 2025-02-15 08:43:37,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:43:37,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:43:37,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:43:37,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:37,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28138.12 MB 2025-02-15 08:43:37,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30379.97 MB 2025-02-15 08:43:37,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:43:37,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34590.43 MB 2025-02-15 08:43:37,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-15 08:43:37,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 08:43:37,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.26 MB 2025-02-15 08:43:37,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:43:37,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:43:37,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:43:37,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:37,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26248.58 MB 2025-02-15 08:43:37,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30379.97 MB 2025-02-15 08:43:37,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:43:37,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34590.43 MB 2025-02-15 08:43:37,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-15 08:43:37,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 08:43:37,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.26 MB 2025-02-15 08:43:38,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:43:38,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:43:38,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:43:38,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:38,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31913.52 MB 2025-02-15 08:43:38,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32680.52 MB 2025-02-15 08:43:38,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:43:38,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39309.02 MB 2025-02-15 08:43:38,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39722.16 MB 2025-02-15 08:43:38,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 08:43:38,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33388.31 MB 2025-02-15 08:43:38,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:43:38,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:43:38,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:43:38,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:38,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33093.41 MB 2025-02-15 08:43:38,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33322.54 MB 2025-02-15 08:43:38,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-15 08:43:38,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39722.16 MB 2025-02-15 08:43:38,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39722.16 MB 2025-02-15 08:43:38,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:43:38,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.05 MB 2025-02-15 08:43:38,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:43:38,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:43:38,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.16 seconds 2025-02-15 08:43:38,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:38,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19630.27 MB 2025-02-15 08:43:38,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33523.59 MB 2025-02-15 08:43:38,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13893.32 MB 2025-02-15 08:43:38,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52787.41 MB 2025-02-15 08:43:38,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39722.16 MB 2025-02-15 08:43:38,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13065.26 MB 2025-02-15 08:43:38,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.05 MB 2025-02-15 08:43:38,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:43:38,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:43:38,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:43:38,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:38,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33523.59 MB 2025-02-15 08:43:38,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24634.28 MB 2025-02-15 08:43:38,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8889.31 MB 2025-02-15 08:43:38,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39722.16 MB 2025-02-15 08:43:38,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39722.16 MB 2025-02-15 08:43:38,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:43:38,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36034.95 MB 2025-02-15 08:43:38,369 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 08:43:38,369 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:43:38,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:43:38,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:43:38,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:43:38,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:43:38,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24634.28 MB 2025-02-15 08:43:38,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33073.11 MB 2025-02-15 08:43:38,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 08:43:38,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39722.16 MB 2025-02-15 08:43:38,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48110.76 MB 2025-02-15 08:43:38,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 08:43:38,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33073.11 MB 2025-02-15 08:43:38,537 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 08:43:38,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:38,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:43:38,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:38,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:43:38,544 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:43:38,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:43:38,545 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:43:38,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:45:49,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:45:49,234 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:45:49,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:45:49,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:45:49,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:45:49,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:45:49,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:46:00,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:46:00,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:46:00,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.97 seconds 2025-02-15 08:46:00,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:00,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17930.04 MB 2025-02-15 08:46:00,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20450.81 MB 2025-02-15 08:46:00,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2520.78 MB 2025-02-15 08:46:00,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56499.37 MB 2025-02-15 08:46:00,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24587.01 MB 2025-02-15 08:46:00,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31912.36 MB 2025-02-15 08:46:00,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29439.84 MB 2025-02-15 08:46:00,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:46:00,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:46:00,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 08:46:00,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:00,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20450.81 MB 2025-02-15 08:46:00,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19480.37 MB 2025-02-15 08:46:00,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -970.45 MB 2025-02-15 08:46:00,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24587.01 MB 2025-02-15 08:46:00,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31314.67 MB 2025-02-15 08:46:00,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6727.66 MB 2025-02-15 08:46:00,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29278.45 MB 2025-02-15 08:46:02,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:46:02,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:46:02,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 08:46:02,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19480.37 MB 2025-02-15 08:46:02,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20011.21 MB 2025-02-15 08:46:02,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:46:02,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31314.67 MB 2025-02-15 08:46:02,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24190.65 MB 2025-02-15 08:46:02,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7124.03 MB 2025-02-15 08:46:02,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23990.79 MB 2025-02-15 08:46:02,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:46:02,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:46:02,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:46:02,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-15 08:46:02,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21900.74 MB 2025-02-15 08:46:02,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:46:02,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24190.65 MB 2025-02-15 08:46:02,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26078.09 MB 2025-02-15 08:46:02,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:46:02,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23318.17 MB 2025-02-15 08:46:02,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:46:02,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:46:02,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:46:02,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21900.74 MB 2025-02-15 08:46:02,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-15 08:46:02,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:46:02,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26078.09 MB 2025-02-15 08:46:02,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31740.40 MB 2025-02-15 08:46:02,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:46:02,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-15 08:46:02,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:46:02,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:46:02,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:46:02,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-15 08:46:02,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-15 08:46:02,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:46:02,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24190.65 MB 2025-02-15 08:46:02,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31740.40 MB 2025-02-15 08:46:02,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 08:46:02,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-15 08:46:02,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:46:02,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:46:02,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:46:02,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25676.14 MB 2025-02-15 08:46:02,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26443.14 MB 2025-02-15 08:46:02,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:46:02,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31740.40 MB 2025-02-15 08:46:02,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32157.73 MB 2025-02-15 08:46:02,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:46:02,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27150.93 MB 2025-02-15 08:46:02,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:46:02,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:46:02,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:46:02,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26856.03 MB 2025-02-15 08:46:02,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27084.57 MB 2025-02-15 08:46:02,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-15 08:46:02,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32157.73 MB 2025-02-15 08:46:02,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32157.73 MB 2025-02-15 08:46:02,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:46:02,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27270.51 MB 2025-02-15 08:46:02,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:46:02,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:46:02,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.35 seconds 2025-02-15 08:46:02,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15449.37 MB 2025-02-15 08:46:02,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27285.65 MB 2025-02-15 08:46:02,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11836.27 MB 2025-02-15 08:46:02,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56499.37 MB 2025-02-15 08:46:02,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32157.73 MB 2025-02-15 08:46:02,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24341.64 MB 2025-02-15 08:46:02,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27285.65 MB 2025-02-15 08:46:02,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:46:02,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:46:02,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:46:02,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27285.65 MB 2025-02-15 08:46:02,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20453.76 MB 2025-02-15 08:46:02,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6831.89 MB 2025-02-15 08:46:02,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32157.73 MB 2025-02-15 08:46:02,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32157.73 MB 2025-02-15 08:46:02,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:46:02,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29797.31 MB 2025-02-15 08:46:02,887 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:46:02,888 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:46:02,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:46:02,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:46:02,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:46:02,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:46:02,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20453.76 MB 2025-02-15 08:46:02,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.78 MB 2025-02-15 08:46:02,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:46:02,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32157.73 MB 2025-02-15 08:46:02,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40548.43 MB 2025-02-15 08:46:02,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 08:46:02,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.78 MB 2025-02-15 08:46:03,065 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:46:03,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:03,066 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:46:03,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:03,067 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:46:03,072 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:46:03,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:03,073 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:46:03,073 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:46:16,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:16,948 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:46:16,953 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:46:16,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:16,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2879, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:46:16,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:46:16,958 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2879, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:47:01,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:47:01,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:47:01,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.73 seconds 2025-02-15 08:47:01,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:01,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33030.72 MB 2025-02-15 08:47:01,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43219.34 MB 2025-02-15 08:47:01,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10188.62 MB 2025-02-15 08:47:01,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73198.99 MB 2025-02-15 08:47:01,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46735.03 MB 2025-02-15 08:47:01,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26463.96 MB 2025-02-15 08:47:01,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53407.96 MB 2025-02-15 08:47:01,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:47:01,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:47:01,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 08:47:01,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:01,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43219.34 MB 2025-02-15 08:47:01,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30746.60 MB 2025-02-15 08:47:01,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12472.74 MB 2025-02-15 08:47:01,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46735.03 MB 2025-02-15 08:47:01,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65445.82 MB 2025-02-15 08:47:01,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18710.79 MB 2025-02-15 08:47:01,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68161.15 MB 2025-02-15 08:47:03,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:47:03,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:47:03,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:47:03,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:03,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30746.60 MB 2025-02-15 08:47:03,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31277.44 MB 2025-02-15 08:47:03,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:47:03,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65445.82 MB 2025-02-15 08:47:03,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33573.31 MB 2025-02-15 08:47:03,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31872.52 MB 2025-02-15 08:47:03,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35255.99 MB 2025-02-15 08:47:03,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:47:03,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:47:03,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:47:03,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:03,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31277.44 MB 2025-02-15 08:47:03,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33166.65 MB 2025-02-15 08:47:03,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-15 08:47:03,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33573.31 MB 2025-02-15 08:47:03,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36404.46 MB 2025-02-15 08:47:03,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:47:03,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34584.08 MB 2025-02-15 08:47:04,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:47:04,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:47:04,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:47:04,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33166.65 MB 2025-02-15 08:47:04,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35408.50 MB 2025-02-15 08:47:04,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:47:04,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36404.46 MB 2025-02-15 08:47:04,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42538.63 MB 2025-02-15 08:47:04,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 08:47:04,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40952.79 MB 2025-02-15 08:47:04,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:47:04,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:47:04,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:47:04,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31277.44 MB 2025-02-15 08:47:04,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35408.50 MB 2025-02-15 08:47:04,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-15 08:47:04,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33573.31 MB 2025-02-15 08:47:04,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42538.63 MB 2025-02-15 08:47:04,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 08:47:04,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40952.79 MB 2025-02-15 08:47:04,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:47:04,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:47:04,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:47:04,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36942.05 MB 2025-02-15 08:47:04,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37709.05 MB 2025-02-15 08:47:04,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:47:04,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42538.63 MB 2025-02-15 08:47:04,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 08:47:04,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:47:04,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38416.84 MB 2025-02-15 08:47:04,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:47:04,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:47:04,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:04,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38121.94 MB 2025-02-15 08:47:04,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38350.96 MB 2025-02-15 08:47:04,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-15 08:47:04,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42953.87 MB 2025-02-15 08:47:04,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 08:47:04,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:47:04,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38566.37 MB 2025-02-15 08:47:04,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:47:04,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:47:04,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.34 seconds 2025-02-15 08:47:04,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22999.71 MB 2025-02-15 08:47:04,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38551.88 MB 2025-02-15 08:47:04,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15552.17 MB 2025-02-15 08:47:04,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63166.22 MB 2025-02-15 08:47:04,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 08:47:04,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20212.35 MB 2025-02-15 08:47:04,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38566.37 MB 2025-02-15 08:47:04,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:47:04,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:47:04,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:47:04,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38551.88 MB 2025-02-15 08:47:04,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28001.82 MB 2025-02-15 08:47:04,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10550.07 MB 2025-02-15 08:47:04,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42953.87 MB 2025-02-15 08:47:04,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 08:47:04,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:47:04,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41061.71 MB 2025-02-15 08:47:04,594 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 08:47:04,594 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:47:04,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:47:04,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:47:04,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:04,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:04,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28001.82 MB 2025-02-15 08:47:04,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36435.11 MB 2025-02-15 08:47:04,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 08:47:04,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42953.87 MB 2025-02-15 08:47:04,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47146.07 MB 2025-02-15 08:47:04,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 08:47:04,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36435.11 MB 2025-02-15 08:47:04,758 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 08:47:04,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:04,760 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:47:04,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:04,760 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:47:04,765 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:47:04,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:04,766 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:47:04,766 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:47:22,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:22,785 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:47:22,793 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:47:22,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:22,800 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:47:22,802 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:22,802 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:47:27,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:47:27,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:47:27,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.55 seconds 2025-02-15 08:47:27,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:27,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14989.47 MB 2025-02-15 08:47:27,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16015.77 MB 2025-02-15 08:47:27,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-15 08:47:27,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55530.49 MB 2025-02-15 08:47:27,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-15 08:47:27,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34819.01 MB 2025-02-15 08:47:27,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24913.83 MB 2025-02-15 08:47:27,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:47:27,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:47:27,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:27,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:27,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16015.77 MB 2025-02-15 08:47:27,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16506.90 MB 2025-02-15 08:47:27,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 491.13 MB 2025-02-15 08:47:27,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-15 08:47:27,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22743.61 MB 2025-02-15 08:47:27,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2032.14 MB 2025-02-15 08:47:27,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20076.06 MB 2025-02-15 08:47:28,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:47:28,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:47:28,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.39 seconds 2025-02-15 08:47:28,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:28,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16506.90 MB 2025-02-15 08:47:28,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16890.43 MB 2025-02-15 08:47:28,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 383.53 MB 2025-02-15 08:47:28,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22743.61 MB 2025-02-15 08:47:28,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20162.02 MB 2025-02-15 08:47:28,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2581.59 MB 2025-02-15 08:47:28,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20847.46 MB 2025-02-15 08:47:28,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:47:28,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:47:28,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:47:28,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:28,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16890.43 MB 2025-02-15 08:47:28,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18255.61 MB 2025-02-15 08:47:28,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1365.18 MB 2025-02-15 08:47:28,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20162.02 MB 2025-02-15 08:47:28,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 08:47:28,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 683.67 MB 2025-02-15 08:47:28,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19279.70 MB 2025-02-15 08:47:28,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:47:28,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:47:28,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 08:47:28,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:28,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18255.61 MB 2025-02-15 08:47:28,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19875.37 MB 2025-02-15 08:47:28,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1619.76 MB 2025-02-15 08:47:28,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20845.69 MB 2025-02-15 08:47:28,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25289.56 MB 2025-02-15 08:47:28,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4443.87 MB 2025-02-15 08:47:28,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23882.01 MB 2025-02-15 08:47:28,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:47:28,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:47:28,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:47:28,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:28,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16890.43 MB 2025-02-15 08:47:28,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19875.37 MB 2025-02-15 08:47:28,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2984.94 MB 2025-02-15 08:47:28,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20162.02 MB 2025-02-15 08:47:28,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25289.56 MB 2025-02-15 08:47:28,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5127.54 MB 2025-02-15 08:47:28,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23882.01 MB 2025-02-15 08:47:29,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:47:29,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:47:29,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:47:29,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:29,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20983.35 MB 2025-02-15 08:47:29,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21537.51 MB 2025-02-15 08:47:29,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 554.16 MB 2025-02-15 08:47:29,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25289.56 MB 2025-02-15 08:47:29,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25589.45 MB 2025-02-15 08:47:29,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 299.89 MB 2025-02-15 08:47:29,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22048.89 MB 2025-02-15 08:47:29,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:47:29,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:47:29,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:47:29,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:29,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21835.83 MB 2025-02-15 08:47:29,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22066.10 MB 2025-02-15 08:47:29,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.27 MB 2025-02-15 08:47:29,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25589.45 MB 2025-02-15 08:47:29,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25591.55 MB 2025-02-15 08:47:29,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 08:47:29,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22169.21 MB 2025-02-15 08:47:29,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:47:29,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:47:29,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.26 seconds 2025-02-15 08:47:29,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:29,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-15 08:47:29,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22267.17 MB 2025-02-15 08:47:29,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8288.08 MB 2025-02-15 08:47:29,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55530.49 MB 2025-02-15 08:47:29,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25591.55 MB 2025-02-15 08:47:29,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29938.94 MB 2025-02-15 08:47:29,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22267.17 MB 2025-02-15 08:47:29,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:47:29,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:47:29,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:47:29,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:29,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22267.17 MB 2025-02-15 08:47:29,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25281.20 MB 2025-02-15 08:47:29,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 08:47:29,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25591.55 MB 2025-02-15 08:47:29,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27067.94 MB 2025-02-15 08:47:29,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-15 08:47:29,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25582.83 MB 2025-02-15 08:47:29,355 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:47:29,355 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 08:47:29,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:47:29,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:47:29,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:29,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:29,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18459.38 MB 2025-02-15 08:47:29,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26898.40 MB 2025-02-15 08:47:29,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:47:29,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-15 08:47:29,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37557.90 MB 2025-02-15 08:47:29,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:47:29,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26898.40 MB 2025-02-15 08:47:29,522 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:47:29,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:29,523 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:47:29,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:29,524 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:47:29,529 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:47:29,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:29,530 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:47:29,530 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 08:47:50,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:50,215 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:47:50,219 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:47:50,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:50,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:47:50,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:50,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:47:54,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:47:54,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:47:54,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.68 seconds 2025-02-15 08:47:54,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:54,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15066.12 MB 2025-02-15 08:47:54,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16131.34 MB 2025-02-15 08:47:54,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1065.22 MB 2025-02-15 08:47:54,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50142.90 MB 2025-02-15 08:47:54,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20849.89 MB 2025-02-15 08:47:54,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29293.02 MB 2025-02-15 08:47:54,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24990.48 MB 2025-02-15 08:47:54,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:47:54,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:47:54,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:54,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:54,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16131.34 MB 2025-02-15 08:47:54,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16647.51 MB 2025-02-15 08:47:54,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.16 MB 2025-02-15 08:47:54,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20849.89 MB 2025-02-15 08:47:54,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22433.23 MB 2025-02-15 08:47:54,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1583.35 MB 2025-02-15 08:47:54,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20359.33 MB 2025-02-15 08:47:56,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:47:56,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:47:56,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.45 seconds 2025-02-15 08:47:56,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16647.51 MB 2025-02-15 08:47:56,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17046.97 MB 2025-02-15 08:47:56,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 399.46 MB 2025-02-15 08:47:56,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22433.23 MB 2025-02-15 08:47:56,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20111.69 MB 2025-02-15 08:47:56,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2321.55 MB 2025-02-15 08:47:56,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20988.06 MB 2025-02-15 08:47:56,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:47:56,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:47:56,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:47:56,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17046.97 MB 2025-02-15 08:47:56,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18470.73 MB 2025-02-15 08:47:56,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1423.77 MB 2025-02-15 08:47:56,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20111.69 MB 2025-02-15 08:47:56,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21533.56 MB 2025-02-15 08:47:56,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1421.87 MB 2025-02-15 08:47:56,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19537.35 MB 2025-02-15 08:47:56,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:47:56,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:47:56,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:47:56,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18470.73 MB 2025-02-15 08:47:56,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20157.75 MB 2025-02-15 08:47:56,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1687.01 MB 2025-02-15 08:47:56,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21533.56 MB 2025-02-15 08:47:56,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26157.78 MB 2025-02-15 08:47:56,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4624.22 MB 2025-02-15 08:47:56,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24332.95 MB 2025-02-15 08:47:56,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:47:56,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:47:56,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:47:56,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17046.97 MB 2025-02-15 08:47:56,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20157.75 MB 2025-02-15 08:47:56,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3110.78 MB 2025-02-15 08:47:56,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20111.69 MB 2025-02-15 08:47:56,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26157.78 MB 2025-02-15 08:47:56,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6046.09 MB 2025-02-15 08:47:56,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24332.95 MB 2025-02-15 08:47:56,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:47:56,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:47:56,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 08:47:56,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21311.74 MB 2025-02-15 08:47:56,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21888.91 MB 2025-02-15 08:47:56,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 577.17 MB 2025-02-15 08:47:56,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26157.78 MB 2025-02-15 08:47:56,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26470.25 MB 2025-02-15 08:47:56,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 312.48 MB 2025-02-15 08:47:56,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22421.52 MB 2025-02-15 08:47:56,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:47:56,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:47:56,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:47:56,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22199.61 MB 2025-02-15 08:47:56,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22416.91 MB 2025-02-15 08:47:56,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.30 MB 2025-02-15 08:47:56,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26470.25 MB 2025-02-15 08:47:56,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26470.25 MB 2025-02-15 08:47:56,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:47:56,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22544.68 MB 2025-02-15 08:47:56,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:47:56,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:47:56,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.47 seconds 2025-02-15 08:47:56,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14017.41 MB 2025-02-15 08:47:56,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22617.98 MB 2025-02-15 08:47:56,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8600.57 MB 2025-02-15 08:47:56,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50142.90 MB 2025-02-15 08:47:56,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26470.25 MB 2025-02-15 08:47:56,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23672.65 MB 2025-02-15 08:47:56,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22617.98 MB 2025-02-15 08:47:56,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:47:56,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:47:56,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:47:56,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22617.98 MB 2025-02-15 08:47:56,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25632.01 MB 2025-02-15 08:47:56,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 08:47:56,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26470.25 MB 2025-02-15 08:47:56,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26872.91 MB 2025-02-15 08:47:56,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 08:47:56,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25933.64 MB 2025-02-15 08:47:56,983 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:47:56,983 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:47:56,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:47:56,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:47:56,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:47:56,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:47:56,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18554.59 MB 2025-02-15 08:47:56,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26993.98 MB 2025-02-15 08:47:56,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.38 MB 2025-02-15 08:47:56,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26872.91 MB 2025-02-15 08:47:56,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37362.86 MB 2025-02-15 08:47:56,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:47:56,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.98 MB 2025-02-15 08:47:57,147 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:47:57,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:57,148 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:47:57,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:57,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:47:57,154 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:47:57,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:47:57,155 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:47:57,155 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:48:08,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:08,612 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:48:08,617 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:48:08,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:08,620 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 462, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:48:08,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:08,621 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 462, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:48:15,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:48:15,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:48:15,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.21 seconds 2025-02-15 08:48:15,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:15,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16188.00 MB 2025-02-15 08:48:15,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17823.77 MB 2025-02-15 08:48:15,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1635.78 MB 2025-02-15 08:48:15,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49947.87 MB 2025-02-15 08:48:15,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21277.70 MB 2025-02-15 08:48:15,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28670.16 MB 2025-02-15 08:48:15,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26792.64 MB 2025-02-15 08:48:15,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:48:15,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:48:15,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 08:48:15,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:15,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17823.77 MB 2025-02-15 08:48:15,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.69 MB 2025-02-15 08:48:15,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.92 MB 2025-02-15 08:48:15,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21277.70 MB 2025-02-15 08:48:15,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26558.33 MB 2025-02-15 08:48:15,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5280.63 MB 2025-02-15 08:48:15,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25203.93 MB 2025-02-15 08:48:17,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:48:17,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:48:17,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:48:17,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:17,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18180.69 MB 2025-02-15 08:48:17,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18711.53 MB 2025-02-15 08:48:17,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:48:17,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26558.33 MB 2025-02-15 08:48:17,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21766.34 MB 2025-02-15 08:48:17,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4791.99 MB 2025-02-15 08:48:17,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22691.12 MB 2025-02-15 08:48:17,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:48:17,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:48:17,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:17,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:17,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-15 08:48:17,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20601.07 MB 2025-02-15 08:48:17,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:48:17,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21766.34 MB 2025-02-15 08:48:17,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24597.50 MB 2025-02-15 08:48:17,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:48:17,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22018.50 MB 2025-02-15 08:48:18,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:48:18,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:48:18,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:48:18,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20601.07 MB 2025-02-15 08:48:18,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-15 08:48:18,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:48:18,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24597.50 MB 2025-02-15 08:48:18,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30259.81 MB 2025-02-15 08:48:18,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:48:18,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-15 08:48:18,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:48:18,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:48:18,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:48:18,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-15 08:48:18,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-15 08:48:18,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:48:18,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21766.34 MB 2025-02-15 08:48:18,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30259.81 MB 2025-02-15 08:48:18,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 08:48:18,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-15 08:48:18,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:48:18,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:48:18,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:48:18,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24376.47 MB 2025-02-15 08:48:18,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25143.47 MB 2025-02-15 08:48:18,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:48:18,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30259.81 MB 2025-02-15 08:48:18,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-15 08:48:18,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:48:18,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25851.26 MB 2025-02-15 08:48:18,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:48:18,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:48:18,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:48:18,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25556.36 MB 2025-02-15 08:48:18,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25784.04 MB 2025-02-15 08:48:18,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.68 MB 2025-02-15 08:48:18,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-15 08:48:18,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-15 08:48:18,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:18,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25992.87 MB 2025-02-15 08:48:18,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:48:18,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:48:18,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.59 seconds 2025-02-15 08:48:18,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-15 08:48:18,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.11 MB 2025-02-15 08:48:18,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11406.76 MB 2025-02-15 08:48:18,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49947.87 MB 2025-02-15 08:48:18,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-15 08:48:18,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19270.73 MB 2025-02-15 08:48:18,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25992.87 MB 2025-02-15 08:48:18,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:48:18,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:48:18,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:48:18,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25985.11 MB 2025-02-15 08:48:18,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19582.74 MB 2025-02-15 08:48:18,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6402.37 MB 2025-02-15 08:48:18,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-15 08:48:18,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-15 08:48:18,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:18,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28496.78 MB 2025-02-15 08:48:18,498 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:48:18,499 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:48:18,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:48:18,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:48:18,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:48:18,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:18,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19582.74 MB 2025-02-15 08:48:18,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28021.76 MB 2025-02-15 08:48:18,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:48:18,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-15 08:48:18,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41167.09 MB 2025-02-15 08:48:18,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 08:48:18,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28021.76 MB 2025-02-15 08:48:18,663 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:48:18,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:18,664 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:48:18,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:18,665 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:48:18,670 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:48:18,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:18,671 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:48:18,671 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:48:26,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:26,817 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:48:26,822 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:48:26,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:26,825 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 207, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:48:26,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:26,826 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 207, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:48:30,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:48:30,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:48:30,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-15 08:48:30,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:30,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14411.12 MB 2025-02-15 08:48:30,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15143.68 MB 2025-02-15 08:48:30,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 732.56 MB 2025-02-15 08:48:30,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53752.10 MB 2025-02-15 08:48:30,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20229.13 MB 2025-02-15 08:48:30,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33522.97 MB 2025-02-15 08:48:30,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24108.98 MB 2025-02-15 08:48:30,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:48:30,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:48:30,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:30,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:30,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15143.68 MB 2025-02-15 08:48:30,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15329.98 MB 2025-02-15 08:48:30,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.31 MB 2025-02-15 08:48:30,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20229.13 MB 2025-02-15 08:48:30,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20229.13 MB 2025-02-15 08:48:30,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:30,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17714.10 MB 2025-02-15 08:48:30,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:48:30,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:48:30,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.89 seconds 2025-02-15 08:48:30,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:30,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15329.98 MB 2025-02-15 08:48:30,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15572.84 MB 2025-02-15 08:48:30,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 242.86 MB 2025-02-15 08:48:30,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20229.13 MB 2025-02-15 08:48:30,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 08:48:30,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 08:48:30,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19500.67 MB 2025-02-15 08:48:30,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:48:30,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:48:30,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:30,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:30,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15572.84 MB 2025-02-15 08:48:30,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16437.09 MB 2025-02-15 08:48:30,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 864.25 MB 2025-02-15 08:48:30,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 08:48:30,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 08:48:30,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:30,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17085.57 MB 2025-02-15 08:48:31,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:48:31,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:48:31,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:48:31,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16437.09 MB 2025-02-15 08:48:31,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.42 MB 2025-02-15 08:48:31,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.32 MB 2025-02-15 08:48:31,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 08:48:31,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21485.32 MB 2025-02-15 08:48:31,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1728.05 MB 2025-02-15 08:48:31,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20000.94 MB 2025-02-15 08:48:31,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:48:31,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:48:31,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:48:31,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15572.84 MB 2025-02-15 08:48:31,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.42 MB 2025-02-15 08:48:31,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1890.58 MB 2025-02-15 08:48:31,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 08:48:31,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21485.32 MB 2025-02-15 08:48:31,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1728.05 MB 2025-02-15 08:48:31,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20000.94 MB 2025-02-15 08:48:31,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:48:31,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:48:31,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:48:31,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18165.01 MB 2025-02-15 08:48:31,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18515.92 MB 2025-02-15 08:48:31,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 350.90 MB 2025-02-15 08:48:31,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21485.32 MB 2025-02-15 08:48:31,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-15 08:48:31,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-15 08:48:31,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18843.36 MB 2025-02-15 08:48:31,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:48:31,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:48:31,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:31,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18704.82 MB 2025-02-15 08:48:31,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18929.72 MB 2025-02-15 08:48:31,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.90 MB 2025-02-15 08:48:31,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-15 08:48:31,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-15 08:48:31,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:31,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18962.87 MB 2025-02-15 08:48:31,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:48:31,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:48:31,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.36 seconds 2025-02-15 08:48:31,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13689.91 MB 2025-02-15 08:48:31,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19130.36 MB 2025-02-15 08:48:31,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5440.44 MB 2025-02-15 08:48:31,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53752.10 MB 2025-02-15 08:48:31,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-15 08:48:31,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32075.94 MB 2025-02-15 08:48:31,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.36 MB 2025-02-15 08:48:31,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:48:31,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:48:31,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:48:31,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19130.36 MB 2025-02-15 08:48:31,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17663.36 MB 2025-02-15 08:48:31,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1467.00 MB 2025-02-15 08:48:31,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-15 08:48:31,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-15 08:48:31,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:31,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.36 MB 2025-02-15 08:48:31,472 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 08:48:31,472 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:48:31,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:48:31,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:48:31,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:48:31,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:31,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17663.36 MB 2025-02-15 08:48:31,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26084.13 MB 2025-02-15 08:48:31,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 08:48:31,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-15 08:48:31,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32140.95 MB 2025-02-15 08:48:31,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 08:48:31,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26084.13 MB 2025-02-15 08:48:31,636 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 08:48:31,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:31,637 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:48:31,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:31,638 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:48:31,643 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:48:31,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:31,644 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:48:31,644 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:48:39,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:39,503 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:48:39,508 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:48:39,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:39,511 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 170, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:48:39,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:39,512 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 170, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:48:42,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:48:42,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:48:42,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.66 seconds 2025-02-15 08:48:42,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:42,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14153.29 MB 2025-02-15 08:48:42,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14754.91 MB 2025-02-15 08:48:42,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.62 MB 2025-02-15 08:48:42,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40512.78 MB 2025-02-15 08:48:42,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19870.52 MB 2025-02-15 08:48:42,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20642.27 MB 2025-02-15 08:48:42,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23624.66 MB 2025-02-15 08:48:42,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:48:42,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:48:42,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:42,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:42,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14754.91 MB 2025-02-15 08:48:42,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14983.19 MB 2025-02-15 08:48:42,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-15 08:48:42,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19870.52 MB 2025-02-15 08:48:42,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19870.52 MB 2025-02-15 08:48:42,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:42,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17016.39 MB 2025-02-15 08:48:42,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:48:42,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:48:42,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 08:48:42,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:42,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14983.19 MB 2025-02-15 08:48:42,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.85 MB 2025-02-15 08:48:42,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 08:48:42,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19870.52 MB 2025-02-15 08:48:42,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19438.50 MB 2025-02-15 08:48:42,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -432.01 MB 2025-02-15 08:48:42,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19153.88 MB 2025-02-15 08:48:42,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:48:42,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:48:42,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:42,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:42,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.79 MB 2025-02-15 08:48:42,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15957.14 MB 2025-02-15 08:48:42,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 08:48:42,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19438.50 MB 2025-02-15 08:48:42,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19438.50 MB 2025-02-15 08:48:42,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:42,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16527.66 MB 2025-02-15 08:48:43,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:48:43,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:48:43,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:48:43,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15957.14 MB 2025-02-15 08:48:43,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16859.53 MB 2025-02-15 08:48:43,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 08:48:43,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19438.50 MB 2025-02-15 08:48:43,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20392.71 MB 2025-02-15 08:48:43,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 954.20 MB 2025-02-15 08:48:43,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19091.98 MB 2025-02-15 08:48:43,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:48:43,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:48:43,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 08:48:43,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.79 MB 2025-02-15 08:48:43,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16859.53 MB 2025-02-15 08:48:43,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 08:48:43,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19438.50 MB 2025-02-15 08:48:43,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20392.71 MB 2025-02-15 08:48:43,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 954.20 MB 2025-02-15 08:48:43,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19091.98 MB 2025-02-15 08:48:43,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:48:43,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:48:43,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 08:48:43,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.78 MB 2025-02-15 08:48:43,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17786.41 MB 2025-02-15 08:48:43,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-15 08:48:43,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20392.71 MB 2025-02-15 08:48:43,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20560.48 MB 2025-02-15 08:48:43,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 08:48:43,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18079.26 MB 2025-02-15 08:48:43,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:48:43,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:48:43,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:48:43,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17952.61 MB 2025-02-15 08:48:43,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.77 MB 2025-02-15 08:48:43,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-15 08:48:43,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20560.48 MB 2025-02-15 08:48:43,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20560.48 MB 2025-02-15 08:48:43,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:43,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18195.62 MB 2025-02-15 08:48:43,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:48:43,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:48:43,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.64 seconds 2025-02-15 08:48:43,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13561.00 MB 2025-02-15 08:48:43,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18381.76 MB 2025-02-15 08:48:43,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4820.76 MB 2025-02-15 08:48:43,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40512.78 MB 2025-02-15 08:48:43,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20560.48 MB 2025-02-15 08:48:43,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19952.30 MB 2025-02-15 08:48:43,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18381.76 MB 2025-02-15 08:48:43,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:48:43,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:48:43,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:48:43,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18381.76 MB 2025-02-15 08:48:43,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17437.25 MB 2025-02-15 08:48:43,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -944.51 MB 2025-02-15 08:48:43,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20560.48 MB 2025-02-15 08:48:43,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20560.48 MB 2025-02-15 08:48:43,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:48:43,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19185.20 MB 2025-02-15 08:48:43,444 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 08:48:43,444 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 08:48:43,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:48:43,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:48:43,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:48:43,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:48:43,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17437.25 MB 2025-02-15 08:48:43,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25872.85 MB 2025-02-15 08:48:43,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 08:48:43,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20560.48 MB 2025-02-15 08:48:43,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28949.09 MB 2025-02-15 08:48:43,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 08:48:43,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25872.85 MB 2025-02-15 08:48:43,609 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 08:48:43,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:43,610 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:48:43,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:43,611 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:48:43,616 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:48:43,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:48:43,617 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:48:43,617 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 08:49:50,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:50,378 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:49:50,383 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:49:50,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:50,387 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:49:50,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:50,388 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:49:52,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:49:52,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:49:52,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.25 seconds 2025-02-15 08:49:52,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:52,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-15 08:49:52,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-15 08:49:52,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-15 08:49:52,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37337.69 MB 2025-02-15 08:49:52,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18104.71 MB 2025-02-15 08:49:52,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19232.98 MB 2025-02-15 08:49:52,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.46 MB 2025-02-15 08:49:52,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:49:52,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:49:52,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:49:52,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:52,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-15 08:49:52,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14740.86 MB 2025-02-15 08:49:52,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.62 MB 2025-02-15 08:49:52,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18104.71 MB 2025-02-15 08:49:52,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18104.71 MB 2025-02-15 08:49:52,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:49:52,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16528.98 MB 2025-02-15 08:49:53,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:49:53,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:49:53,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 08:49:53,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14740.86 MB 2025-02-15 08:49:53,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14933.29 MB 2025-02-15 08:49:53,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 08:49:53,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18104.71 MB 2025-02-15 08:49:53,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17723.03 MB 2025-02-15 08:49:53,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -381.68 MB 2025-02-15 08:49:53,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18911.54 MB 2025-02-15 08:49:53,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:49:53,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:49:53,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:49:53,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-15 08:49:53,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15618.01 MB 2025-02-15 08:49:53,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 08:49:53,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17723.03 MB 2025-02-15 08:49:53,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17723.03 MB 2025-02-15 08:49:53,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:49:53,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16131.83 MB 2025-02-15 08:49:53,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:49:53,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:49:53,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 08:49:53,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15618.01 MB 2025-02-15 08:49:53,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-15 08:49:53,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 08:49:53,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17723.03 MB 2025-02-15 08:49:53,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19270.73 MB 2025-02-15 08:49:53,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1547.70 MB 2025-02-15 08:49:53,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18441.40 MB 2025-02-15 08:49:53,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:49:53,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:49:53,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:49:53,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-15 08:49:53,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-15 08:49:53,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 08:49:53,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17723.03 MB 2025-02-15 08:49:53,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19270.73 MB 2025-02-15 08:49:53,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1547.70 MB 2025-02-15 08:49:53,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18441.40 MB 2025-02-15 08:49:53,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:49:53,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:49:53,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 08:49:53,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16987.55 MB 2025-02-15 08:49:53,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17265.59 MB 2025-02-15 08:49:53,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 08:49:53,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19270.73 MB 2025-02-15 08:49:53,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-15 08:49:53,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-15 08:49:53,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17531.70 MB 2025-02-15 08:49:53,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:49:53,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:49:53,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:49:53,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17415.27 MB 2025-02-15 08:49:53,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17643.18 MB 2025-02-15 08:49:53,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.91 MB 2025-02-15 08:49:53,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19419.63 MB 2025-02-15 08:49:53,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-15 08:49:53,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:49:53,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17646.98 MB 2025-02-15 08:49:53,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:49:53,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:49:53,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.13 seconds 2025-02-15 08:49:53,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-15 08:49:53,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17844.00 MB 2025-02-15 08:49:53,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4370.11 MB 2025-02-15 08:49:53,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37337.69 MB 2025-02-15 08:49:53,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-15 08:49:53,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17918.07 MB 2025-02-15 08:49:53,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17844.00 MB 2025-02-15 08:49:53,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:49:53,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:49:53,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:49:53,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.00 MB 2025-02-15 08:49:53,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17272.00 MB 2025-02-15 08:49:53,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -572.00 MB 2025-02-15 08:49:53,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19419.63 MB 2025-02-15 08:49:53,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19688.06 MB 2025-02-15 08:49:53,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-15 08:49:53,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18948.21 MB 2025-02-15 08:49:53,801 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 08:49:53,801 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-15 08:49:53,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:49:53,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:49:53,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:49:53,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:49:53,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17272.00 MB 2025-02-15 08:49:53,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25701.12 MB 2025-02-15 08:49:53,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 08:49:53,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19688.06 MB 2025-02-15 08:49:53,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30163.34 MB 2025-02-15 08:49:53,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 08:49:53,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25701.12 MB 2025-02-15 08:49:53,965 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 08:49:53,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:53,966 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:49:53,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:53,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:49:53,971 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:49:53,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:49:53,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:49:53,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-15 08:50:02,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:02,981 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:50:02,986 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:50:02,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:02,989 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1592, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:50:02,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:02,990 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1592, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:50:27,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:50:27,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:50:27,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.62 seconds 2025-02-15 08:50:27,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:27,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24062.02 MB 2025-02-15 08:50:27,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29697.06 MB 2025-02-15 08:50:27,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5635.05 MB 2025-02-15 08:50:27,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38543.56 MB 2025-02-15 08:50:27,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38872.81 MB 2025-02-15 08:50:27,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 329.25 MB 2025-02-15 08:50:27,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38516.22 MB 2025-02-15 08:50:27,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:50:27,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:50:27,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:50:27,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:27,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29697.06 MB 2025-02-15 08:50:27,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24054.16 MB 2025-02-15 08:50:27,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5642.90 MB 2025-02-15 08:50:27,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38872.81 MB 2025-02-15 08:50:27,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49549.41 MB 2025-02-15 08:50:27,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10676.60 MB 2025-02-15 08:50:27,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45755.89 MB 2025-02-15 08:50:29,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:50:29,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:50:29,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:50:29,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:29,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24054.16 MB 2025-02-15 08:50:29,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24585.00 MB 2025-02-15 08:50:29,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:50:29,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49549.41 MB 2025-02-15 08:50:29,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33237.76 MB 2025-02-15 08:50:29,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16311.65 MB 2025-02-15 08:50:29,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28563.55 MB 2025-02-15 08:50:29,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:50:29,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:50:29,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:50:29,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:29,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-15 08:50:29,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26474.54 MB 2025-02-15 08:50:29,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:50:29,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 08:50:29,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33237.76 MB 2025-02-15 08:50:29,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:50:29,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27891.97 MB 2025-02-15 08:50:29,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:50:29,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:50:29,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:50:29,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:29,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26474.54 MB 2025-02-15 08:50:29,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-15 08:50:29,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:50:29,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 08:50:29,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-15 08:50:29,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:50:29,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-15 08:50:29,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:50:29,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:50:29,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:50:29,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:29,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-15 08:50:29,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-15 08:50:29,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:50:29,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-15 08:50:29,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-15 08:50:29,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 08:50:29,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-15 08:50:30,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:50:30,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:50:30,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:50:30,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:30,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30249.94 MB 2025-02-15 08:50:30,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31016.94 MB 2025-02-15 08:50:30,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:50:30,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-15 08:50:30,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-15 08:50:30,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:50:30,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31724.73 MB 2025-02-15 08:50:30,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:50:30,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:50:30,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:50:30,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:30,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.83 MB 2025-02-15 08:50:30,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31657.57 MB 2025-02-15 08:50:30,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.74 MB 2025-02-15 08:50:30,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37429.97 MB 2025-02-15 08:50:30,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-15 08:50:30,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:50:30,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31867.23 MB 2025-02-15 08:50:30,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:50:30,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:50:30,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.08 seconds 2025-02-15 08:50:30,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:30,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18515.36 MB 2025-02-15 08:50:30,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31857.81 MB 2025-02-15 08:50:30,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13342.45 MB 2025-02-15 08:50:30,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38543.56 MB 2025-02-15 08:50:30,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-15 08:50:30,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1113.59 MB 2025-02-15 08:50:30,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31867.23 MB 2025-02-15 08:50:30,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:50:30,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:50:30,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:50:30,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:30,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31857.81 MB 2025-02-15 08:50:30,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23507.41 MB 2025-02-15 08:50:30,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8350.39 MB 2025-02-15 08:50:30,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37429.97 MB 2025-02-15 08:50:30,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-15 08:50:30,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:50:30,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34359.03 MB 2025-02-15 08:50:30,358 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 08:50:30,358 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:50:30,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:50:30,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:50:30,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:50:30,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:50:30,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23507.41 MB 2025-02-15 08:50:30,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31911.01 MB 2025-02-15 08:50:30,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.60 MB 2025-02-15 08:50:30,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37429.97 MB 2025-02-15 08:50:30,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-15 08:50:30,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:50:30,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31911.01 MB 2025-02-15 08:50:30,525 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 08:50:30,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:30,526 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:50:30,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:30,527 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:50:30,532 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:50:30,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:50:30,533 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:50:30,533 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:52:02,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:02,406 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:52:02,411 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:52:02,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:02,416 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:52:02,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:02,417 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:52:04,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:52:04,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:52:04,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.44 seconds 2025-02-15 08:52:04,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:04,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14069.68 MB 2025-02-15 08:52:04,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14628.83 MB 2025-02-15 08:52:04,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-15 08:52:04,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45785.02 MB 2025-02-15 08:52:04,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 08:52:04,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28363.98 MB 2025-02-15 08:52:04,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23541.05 MB 2025-02-15 08:52:04,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:52:04,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:52:04,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:52:04,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:04,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14628.83 MB 2025-02-15 08:52:04,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14809.09 MB 2025-02-15 08:52:04,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 180.26 MB 2025-02-15 08:52:04,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 08:52:04,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17934.84 MB 2025-02-15 08:52:04,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 513.80 MB 2025-02-15 08:52:04,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16666.23 MB 2025-02-15 08:52:05,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:52:05,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:52:05,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 08:52:05,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14809.09 MB 2025-02-15 08:52:05,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15001.52 MB 2025-02-15 08:52:05,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 08:52:05,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17934.84 MB 2025-02-15 08:52:05,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17934.84 MB 2025-02-15 08:52:05,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:05,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18979.78 MB 2025-02-15 08:52:05,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:52:05,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:52:05,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 08:52:05,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15001.46 MB 2025-02-15 08:52:05,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15686.25 MB 2025-02-15 08:52:05,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 08:52:05,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17934.84 MB 2025-02-15 08:52:05,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17934.84 MB 2025-02-15 08:52:05,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:05,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16200.07 MB 2025-02-15 08:52:05,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:52:05,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:52:05,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 08:52:05,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15686.25 MB 2025-02-15 08:52:05,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16498.96 MB 2025-02-15 08:52:05,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 08:52:05,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17934.84 MB 2025-02-15 08:52:05,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19654.51 MB 2025-02-15 08:52:05,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-15 08:52:05,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18508.72 MB 2025-02-15 08:52:05,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:52:05,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:52:05,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:52:05,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15001.46 MB 2025-02-15 08:52:05,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16498.96 MB 2025-02-15 08:52:05,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 08:52:05,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17934.84 MB 2025-02-15 08:52:05,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19654.51 MB 2025-02-15 08:52:05,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-15 08:52:05,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18508.72 MB 2025-02-15 08:52:05,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:52:05,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:52:05,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 08:52:05,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17054.87 MB 2025-02-15 08:52:05,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17332.91 MB 2025-02-15 08:52:05,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 08:52:05,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19654.51 MB 2025-02-15 08:52:05,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19805.50 MB 2025-02-15 08:52:05,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 08:52:05,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17599.57 MB 2025-02-15 08:52:05,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:52:05,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:52:05,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:52:05,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17482.59 MB 2025-02-15 08:52:05,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17710.31 MB 2025-02-15 08:52:05,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-15 08:52:05,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19805.50 MB 2025-02-15 08:52:05,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19805.50 MB 2025-02-15 08:52:05,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:05,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17716.44 MB 2025-02-15 08:52:05,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:52:05,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:52:05,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.32 seconds 2025-02-15 08:52:05,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:05,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13519.19 MB 2025-02-15 08:52:05,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17911.09 MB 2025-02-15 08:52:05,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4391.90 MB 2025-02-15 08:52:05,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45785.02 MB 2025-02-15 08:52:05,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19805.50 MB 2025-02-15 08:52:05,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25979.52 MB 2025-02-15 08:52:05,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17911.09 MB 2025-02-15 08:52:06,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:52:06,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:52:06,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 08:52:06,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:06,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17911.09 MB 2025-02-15 08:52:06,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17315.66 MB 2025-02-15 08:52:06,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -595.43 MB 2025-02-15 08:52:06,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19805.50 MB 2025-02-15 08:52:06,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19805.50 MB 2025-02-15 08:52:06,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:06,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19014.60 MB 2025-02-15 08:52:06,022 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 08:52:06,022 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:52:06,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:52:06,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:52:06,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:52:06,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:06,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17315.66 MB 2025-02-15 08:52:06,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25742.16 MB 2025-02-15 08:52:06,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 08:52:06,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19805.50 MB 2025-02-15 08:52:06,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30278.68 MB 2025-02-15 08:52:06,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 08:52:06,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25742.16 MB 2025-02-15 08:52:06,189 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 08:52:06,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:06,190 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:52:06,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:06,191 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:52:06,196 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:52:06,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:06,197 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:52:06,197 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:52:16,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:16,503 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:52:16,509 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:52:16,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:16,512 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1990, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:52:16,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:16,513 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1990, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:52:47,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:52:47,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:52:47,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.75 seconds 2025-02-15 08:52:47,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:47,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26835.35 MB 2025-02-15 08:52:47,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33877.84 MB 2025-02-15 08:52:47,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7042.50 MB 2025-02-15 08:52:47,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42844.82 MB 2025-02-15 08:52:47,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40277.90 MB 2025-02-15 08:52:47,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2566.91 MB 2025-02-15 08:52:47,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42875.00 MB 2025-02-15 08:52:47,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:52:47,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:52:47,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:52:47,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:47,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33877.84 MB 2025-02-15 08:52:47,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26123.24 MB 2025-02-15 08:52:47,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7754.60 MB 2025-02-15 08:52:47,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40277.90 MB 2025-02-15 08:52:47,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54827.94 MB 2025-02-15 08:52:47,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14550.04 MB 2025-02-15 08:52:47,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53632.60 MB 2025-02-15 08:52:49,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:52:49,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:52:49,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 08:52:49,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26123.24 MB 2025-02-15 08:52:49,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26654.08 MB 2025-02-15 08:52:49,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:52:49,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54827.94 MB 2025-02-15 08:52:49,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30461.13 MB 2025-02-15 08:52:49,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24366.81 MB 2025-02-15 08:52:49,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30633.67 MB 2025-02-15 08:52:49,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:52:49,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:52:49,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:52:49,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26654.08 MB 2025-02-15 08:52:49,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28543.62 MB 2025-02-15 08:52:49,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:52:49,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30461.13 MB 2025-02-15 08:52:49,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32348.57 MB 2025-02-15 08:52:49,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:52:49,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29961.04 MB 2025-02-15 08:52:49,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:52:49,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:52:49,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:52:49,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28543.62 MB 2025-02-15 08:52:49,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30785.47 MB 2025-02-15 08:52:49,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:52:49,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32348.57 MB 2025-02-15 08:52:49,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38010.88 MB 2025-02-15 08:52:49,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:52:49,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36329.75 MB 2025-02-15 08:52:49,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:52:49,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:52:49,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:52:49,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26654.08 MB 2025-02-15 08:52:49,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30785.47 MB 2025-02-15 08:52:49,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:52:49,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30461.13 MB 2025-02-15 08:52:49,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38010.88 MB 2025-02-15 08:52:49,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 08:52:49,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36329.75 MB 2025-02-15 08:52:49,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:52:49,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:52:49,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:52:49,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32319.01 MB 2025-02-15 08:52:49,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33086.02 MB 2025-02-15 08:52:49,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:52:49,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38010.88 MB 2025-02-15 08:52:49,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 08:52:49,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:52:49,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33793.80 MB 2025-02-15 08:52:49,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:52:49,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:52:49,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:52:49,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33498.91 MB 2025-02-15 08:52:49,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33727.55 MB 2025-02-15 08:52:49,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-15 08:52:49,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 08:52:49,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 08:52:49,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:49,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33939.66 MB 2025-02-15 08:52:49,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:52:49,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:52:49,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.26 seconds 2025-02-15 08:52:49,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:49,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19902.03 MB 2025-02-15 08:52:49,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33928.62 MB 2025-02-15 08:52:49,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14026.59 MB 2025-02-15 08:52:49,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42844.82 MB 2025-02-15 08:52:49,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 08:52:49,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4418.70 MB 2025-02-15 08:52:49,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33939.66 MB 2025-02-15 08:52:50,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:52:50,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:52:50,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:52:50,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:50,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33928.62 MB 2025-02-15 08:52:50,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24906.42 MB 2025-02-15 08:52:50,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9022.20 MB 2025-02-15 08:52:50,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 08:52:50,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38426.12 MB 2025-02-15 08:52:50,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:52:50,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36440.29 MB 2025-02-15 08:52:50,067 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:52:50,067 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:52:50,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:52:50,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:52:50,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:52:50,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:52:50,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24906.42 MB 2025-02-15 08:52:50,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33345.44 MB 2025-02-15 08:52:50,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:52:50,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38426.12 MB 2025-02-15 08:52:50,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46816.82 MB 2025-02-15 08:52:50,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 08:52:50,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33345.44 MB 2025-02-15 08:52:50,235 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:52:50,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:50,237 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:52:50,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:50,238 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:52:50,242 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:52:50,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:52:50,243 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:52:50,244 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:53:22,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:22,855 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:53:22,860 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:53:22,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:22,863 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:53:22,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:22,864 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:53:25,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:53:25,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:53:25,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.08 seconds 2025-02-15 08:53:25,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:25,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14341.43 MB 2025-02-15 08:53:25,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15038.61 MB 2025-02-15 08:53:25,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 697.17 MB 2025-02-15 08:53:25,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59401.83 MB 2025-02-15 08:53:25,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-15 08:53:25,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38736.49 MB 2025-02-15 08:53:25,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24039.30 MB 2025-02-15 08:53:25,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:53:25,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:53:25,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:53:25,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:25,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15038.61 MB 2025-02-15 08:53:25,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15256.99 MB 2025-02-15 08:53:25,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.39 MB 2025-02-15 08:53:25,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-15 08:53:25,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-15 08:53:25,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:25,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17566.96 MB 2025-02-15 08:53:26,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:53:26,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:53:26,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-15 08:53:26,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:26,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15256.99 MB 2025-02-15 08:53:26,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15495.87 MB 2025-02-15 08:53:26,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 238.88 MB 2025-02-15 08:53:26,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-15 08:53:26,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-15 08:53:26,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:26,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19426.64 MB 2025-02-15 08:53:26,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:53:26,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:53:26,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:53:26,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:26,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15495.81 MB 2025-02-15 08:53:26,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16345.89 MB 2025-02-15 08:53:26,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 850.08 MB 2025-02-15 08:53:26,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-15 08:53:26,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-15 08:53:26,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:26,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16983.74 MB 2025-02-15 08:53:26,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:53:26,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:53:26,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:53:26,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:26,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16345.89 MB 2025-02-15 08:53:26,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17354.76 MB 2025-02-15 08:53:26,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1008.87 MB 2025-02-15 08:53:26,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-15 08:53:26,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20879.25 MB 2025-02-15 08:53:26,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 08:53:26,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19849.65 MB 2025-02-15 08:53:26,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:53:26,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:53:26,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:53:26,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:26,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15495.81 MB 2025-02-15 08:53:26,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17354.76 MB 2025-02-15 08:53:26,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1858.95 MB 2025-02-15 08:53:26,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-15 08:53:26,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20879.25 MB 2025-02-15 08:53:26,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 08:53:26,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19849.65 MB 2025-02-15 08:53:27,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:53:27,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:53:27,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:53:27,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:27,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18044.85 MB 2025-02-15 08:53:27,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18390.01 MB 2025-02-15 08:53:27,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 345.15 MB 2025-02-15 08:53:27,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20879.25 MB 2025-02-15 08:53:27,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21065.89 MB 2025-02-15 08:53:27,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-15 08:53:27,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18714.48 MB 2025-02-15 08:53:27,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:53:27,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:53:27,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:53:27,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:27,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18575.81 MB 2025-02-15 08:53:27,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18790.25 MB 2025-02-15 08:53:27,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.44 MB 2025-02-15 08:53:27,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21065.89 MB 2025-02-15 08:53:27,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21065.89 MB 2025-02-15 08:53:27,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:27,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18828.16 MB 2025-02-15 08:53:27,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:53:27,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:53:27,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.15 seconds 2025-02-15 08:53:27,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:27,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13655.07 MB 2025-02-15 08:53:27,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18991.00 MB 2025-02-15 08:53:27,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5335.93 MB 2025-02-15 08:53:27,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59401.83 MB 2025-02-15 08:53:27,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21065.89 MB 2025-02-15 08:53:27,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38335.94 MB 2025-02-15 08:53:27,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18991.00 MB 2025-02-15 08:53:27,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:53:27,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:53:27,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 08:53:27,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:27,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18991.00 MB 2025-02-15 08:53:27,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17615.74 MB 2025-02-15 08:53:27,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1375.26 MB 2025-02-15 08:53:27,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21065.89 MB 2025-02-15 08:53:27,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21065.89 MB 2025-02-15 08:53:27,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:27,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18991.01 MB 2025-02-15 08:53:27,299 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 08:53:27,300 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 08:53:27,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:53:27,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:53:27,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:53:27,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:27,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17615.74 MB 2025-02-15 08:53:27,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26041.92 MB 2025-02-15 08:53:27,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 08:53:27,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21065.89 MB 2025-02-15 08:53:27,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31536.97 MB 2025-02-15 08:53:27,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 08:53:27,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26041.92 MB 2025-02-15 08:53:27,469 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 08:53:27,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:27,470 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:53:27,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:27,471 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:53:27,476 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:53:27,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:27,477 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:53:27,477 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 08:53:39,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:39,778 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:53:39,784 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:53:39,787 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:39,787 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 811, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:53:39,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:39,789 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 811, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:53:52,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:53:52,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:53:52,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.58 seconds 2025-02-15 08:53:52,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:52,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18619.88 MB 2025-02-15 08:53:52,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21490.89 MB 2025-02-15 08:53:52,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2871.00 MB 2025-02-15 08:53:52,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39913.00 MB 2025-02-15 08:53:52,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25631.39 MB 2025-02-15 08:53:52,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14281.61 MB 2025-02-15 08:53:52,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30356.99 MB 2025-02-15 08:53:52,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:53:52,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:53:52,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 08:53:52,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:52,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21490.89 MB 2025-02-15 08:53:52,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19993.99 MB 2025-02-15 08:53:52,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1496.90 MB 2025-02-15 08:53:52,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25631.39 MB 2025-02-15 08:53:52,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31297.90 MB 2025-02-15 08:53:52,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5666.50 MB 2025-02-15 08:53:52,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29965.48 MB 2025-02-15 08:53:54,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:53:54,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:53:54,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:53:54,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19993.99 MB 2025-02-15 08:53:54,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20524.83 MB 2025-02-15 08:53:54,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:53:54,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31297.90 MB 2025-02-15 08:53:54,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24175.97 MB 2025-02-15 08:53:54,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7121.93 MB 2025-02-15 08:53:54,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24504.41 MB 2025-02-15 08:53:54,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:53:54,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:53:54,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:53:54,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-15 08:53:54,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22414.36 MB 2025-02-15 08:53:54,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:53:54,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 08:53:54,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26063.41 MB 2025-02-15 08:53:54,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:53:54,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23831.79 MB 2025-02-15 08:53:54,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:53:54,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:53:54,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:53:54,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22414.36 MB 2025-02-15 08:53:54,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-15 08:53:54,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:53:54,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26063.41 MB 2025-02-15 08:53:54,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32434.55 MB 2025-02-15 08:53:54,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 08:53:54,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30202.10 MB 2025-02-15 08:53:54,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:53:54,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:53:54,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:53:54,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-15 08:53:54,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-15 08:53:54,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:53:54,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 08:53:54,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32434.55 MB 2025-02-15 08:53:54,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8258.58 MB 2025-02-15 08:53:54,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30202.10 MB 2025-02-15 08:53:54,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:53:54,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:53:54,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:53:54,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26190.31 MB 2025-02-15 08:53:54,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26957.31 MB 2025-02-15 08:53:54,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:53:54,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32434.55 MB 2025-02-15 08:53:54,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-15 08:53:54,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:53:54,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27665.10 MB 2025-02-15 08:53:54,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:53:54,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:53:54,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:53:54,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27370.20 MB 2025-02-15 08:53:54,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27601.78 MB 2025-02-15 08:53:54,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.58 MB 2025-02-15 08:53:54,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-15 08:53:54,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-15 08:53:54,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:54,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27797.22 MB 2025-02-15 08:53:54,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:53:54,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:53:54,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.96 seconds 2025-02-15 08:53:54,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:54,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15794.30 MB 2025-02-15 08:53:54,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27802.85 MB 2025-02-15 08:53:54,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12008.56 MB 2025-02-15 08:53:54,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39913.00 MB 2025-02-15 08:53:54,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-15 08:53:54,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7061.11 MB 2025-02-15 08:53:54,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27802.85 MB 2025-02-15 08:53:55,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:53:55,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:53:55,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:53:55,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:55,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27802.85 MB 2025-02-15 08:53:55,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20799.23 MB 2025-02-15 08:53:55,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7003.62 MB 2025-02-15 08:53:55,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-15 08:53:55,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-15 08:53:55,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:53:55,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30314.52 MB 2025-02-15 08:53:55,037 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 08:53:55,038 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:53:55,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:53:55,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:53:55,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:53:55,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:53:55,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20799.23 MB 2025-02-15 08:53:55,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29238.26 MB 2025-02-15 08:53:55,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 08:53:55,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-15 08:53:55,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41242.59 MB 2025-02-15 08:53:55,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 08:53:55,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29238.26 MB 2025-02-15 08:53:55,201 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 08:53:55,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:55,202 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:53:55,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:55,203 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:53:55,208 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:53:55,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:53:55,209 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:53:55,209 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:55:07,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:07,903 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:55:07,908 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:55:07,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:07,912 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 228, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:55:07,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:07,913 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 228, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:55:11,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:55:11,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:55:11,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.53 seconds 2025-02-15 08:55:11,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:11,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14557.45 MB 2025-02-15 08:55:11,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15364.33 MB 2025-02-15 08:55:11,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 806.88 MB 2025-02-15 08:55:11,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53827.60 MB 2025-02-15 08:55:11,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18838.72 MB 2025-02-15 08:55:11,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34988.88 MB 2025-02-15 08:55:11,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24255.31 MB 2025-02-15 08:55:11,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:55:11,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:55:11,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:55:11,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:11,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15364.33 MB 2025-02-15 08:55:11,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15390.06 MB 2025-02-15 08:55:11,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.73 MB 2025-02-15 08:55:11,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18838.72 MB 2025-02-15 08:55:11,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19461.57 MB 2025-02-15 08:55:11,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 622.85 MB 2025-02-15 08:55:11,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17836.49 MB 2025-02-15 08:55:12,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:55:12,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:55:12,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.89 seconds 2025-02-15 08:55:12,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15390.06 MB 2025-02-15 08:55:12,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15623.63 MB 2025-02-15 08:55:12,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-15 08:55:12,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19461.57 MB 2025-02-15 08:55:12,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19291.70 MB 2025-02-15 08:55:12,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -169.87 MB 2025-02-15 08:55:12,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19560.75 MB 2025-02-15 08:55:12,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:55:12,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:55:12,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:55:12,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.56 MB 2025-02-15 08:55:12,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16454.76 MB 2025-02-15 08:55:12,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-15 08:55:12,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19291.70 MB 2025-02-15 08:55:12,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19291.70 MB 2025-02-15 08:55:12,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:55:12,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17078.43 MB 2025-02-15 08:55:12,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:55:12,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:55:12,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:55:12,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16454.76 MB 2025-02-15 08:55:12,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.21 MB 2025-02-15 08:55:12,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-15 08:55:12,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19291.70 MB 2025-02-15 08:55:12,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21367.88 MB 2025-02-15 08:55:12,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 08:55:12,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.66 MB 2025-02-15 08:55:12,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:55:12,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:55:12,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 08:55:12,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.56 MB 2025-02-15 08:55:12,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.21 MB 2025-02-15 08:55:12,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-15 08:55:12,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19291.70 MB 2025-02-15 08:55:12,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21367.88 MB 2025-02-15 08:55:12,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 08:55:12,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.66 MB 2025-02-15 08:55:12,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:55:12,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:55:12,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 08:55:12,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18115.97 MB 2025-02-15 08:55:12,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18453.45 MB 2025-02-15 08:55:12,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-15 08:55:12,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21367.88 MB 2025-02-15 08:55:12,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 08:55:12,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 08:55:12,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18769.72 MB 2025-02-15 08:55:12,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:55:12,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:55:12,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:55:12,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18635.13 MB 2025-02-15 08:55:12,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18862.62 MB 2025-02-15 08:55:12,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.49 MB 2025-02-15 08:55:12,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21548.24 MB 2025-02-15 08:55:12,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 08:55:12,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:55:12,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18885.80 MB 2025-02-15 08:55:12,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:55:12,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:55:12,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.63 seconds 2025-02-15 08:55:12,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13763.08 MB 2025-02-15 08:55:12,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19063.50 MB 2025-02-15 08:55:12,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5300.42 MB 2025-02-15 08:55:12,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53827.60 MB 2025-02-15 08:55:12,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 08:55:12,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32279.36 MB 2025-02-15 08:55:12,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19063.50 MB 2025-02-15 08:55:12,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:55:12,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:55:12,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:55:12,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19063.50 MB 2025-02-15 08:55:12,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17707.69 MB 2025-02-15 08:55:12,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1355.81 MB 2025-02-15 08:55:12,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21548.24 MB 2025-02-15 08:55:12,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 08:55:12,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:55:12,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19298.37 MB 2025-02-15 08:55:12,831 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 08:55:12,831 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:55:12,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:55:12,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:55:12,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:55:12,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:55:12,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17707.69 MB 2025-02-15 08:55:12,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26138.36 MB 2025-02-15 08:55:12,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 08:55:12,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21548.24 MB 2025-02-15 08:55:12,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32027.71 MB 2025-02-15 08:55:12,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-15 08:55:12,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26138.36 MB 2025-02-15 08:55:13,000 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 08:55:13,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:13,002 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:55:13,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:13,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:55:13,007 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:55:13,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:13,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:55:13,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:55:58,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:58,643 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:55:58,648 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:55:58,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:58,652 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1774, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:55:58,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:55:58,653 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1774, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:56:26,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:56:26,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:56:26,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.34 seconds 2025-02-15 08:56:26,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:26,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25330.22 MB 2025-02-15 08:56:26,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31609.10 MB 2025-02-15 08:56:26,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6278.87 MB 2025-02-15 08:56:26,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44600.13 MB 2025-02-15 08:56:26,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39518.73 MB 2025-02-15 08:56:26,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5081.40 MB 2025-02-15 08:56:26,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40463.90 MB 2025-02-15 08:56:26,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:56:26,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:56:26,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 08:56:26,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:26,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31609.10 MB 2025-02-15 08:56:26,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25000.32 MB 2025-02-15 08:56:26,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6608.77 MB 2025-02-15 08:56:26,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39518.73 MB 2025-02-15 08:56:26,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52495.91 MB 2025-02-15 08:56:26,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12977.18 MB 2025-02-15 08:56:26,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48997.39 MB 2025-02-15 08:56:28,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:56:28,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:56:28,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:56:28,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25000.32 MB 2025-02-15 08:56:28,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25531.16 MB 2025-02-15 08:56:28,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:56:28,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52495.91 MB 2025-02-15 08:56:28,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-15 08:56:28,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17840.47 MB 2025-02-15 08:56:28,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29509.71 MB 2025-02-15 08:56:28,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:56:28,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:56:28,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:56:28,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25531.16 MB 2025-02-15 08:56:28,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27420.70 MB 2025-02-15 08:56:28,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:56:28,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-15 08:56:28,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-15 08:56:28,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:56:28,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28838.13 MB 2025-02-15 08:56:28,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:56:28,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:56:28,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:56:28,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27420.70 MB 2025-02-15 08:56:28,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29662.55 MB 2025-02-15 08:56:28,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:56:28,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-15 08:56:28,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37486.59 MB 2025-02-15 08:56:28,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:56:28,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35206.84 MB 2025-02-15 08:56:28,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:56:28,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:56:28,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:56:28,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25531.16 MB 2025-02-15 08:56:28,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29662.55 MB 2025-02-15 08:56:28,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:56:28,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-15 08:56:28,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37486.59 MB 2025-02-15 08:56:28,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 08:56:28,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35206.84 MB 2025-02-15 08:56:28,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:56:28,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:56:28,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 08:56:28,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31196.10 MB 2025-02-15 08:56:28,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31963.10 MB 2025-02-15 08:56:28,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:56:28,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37486.59 MB 2025-02-15 08:56:28,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37903.93 MB 2025-02-15 08:56:28,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:56:28,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32670.89 MB 2025-02-15 08:56:28,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:56:28,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:56:28,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:56:28,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32375.99 MB 2025-02-15 08:56:28,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32603.99 MB 2025-02-15 08:56:28,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-15 08:56:28,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37903.93 MB 2025-02-15 08:56:28,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37903.93 MB 2025-02-15 08:56:28,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:56:28,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32819.47 MB 2025-02-15 08:56:28,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:56:28,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:56:28,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.82 seconds 2025-02-15 08:56:28,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19149.46 MB 2025-02-15 08:56:28,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32804.84 MB 2025-02-15 08:56:28,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13655.38 MB 2025-02-15 08:56:28,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44600.13 MB 2025-02-15 08:56:28,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37903.93 MB 2025-02-15 08:56:28,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6696.21 MB 2025-02-15 08:56:28,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32819.47 MB 2025-02-15 08:56:28,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:56:28,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:56:28,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:56:28,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32804.84 MB 2025-02-15 08:56:28,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24136.88 MB 2025-02-15 08:56:28,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8667.96 MB 2025-02-15 08:56:28,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37903.93 MB 2025-02-15 08:56:28,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37903.93 MB 2025-02-15 08:56:28,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:56:28,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35302.07 MB 2025-02-15 08:56:28,766 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 08:56:28,766 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:56:28,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:56:28,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:56:28,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:56:28,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:56:28,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24136.88 MB 2025-02-15 08:56:28,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32527.92 MB 2025-02-15 08:56:28,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 08:56:28,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37903.93 MB 2025-02-15 08:56:28,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-15 08:56:28,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 08:56:28,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32527.92 MB 2025-02-15 08:56:28,933 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 08:56:28,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:56:28,935 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:56:28,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:56:28,936 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:56:28,940 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:56:28,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:56:28,941 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:56:28,942 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:57:15,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:15,024 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:57:15,029 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:57:15,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:15,032 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1198, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:57:15,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:15,033 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1198, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:57:33,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:57:33,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:57:33,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.53 seconds 2025-02-15 08:57:33,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:33,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21316.56 MB 2025-02-15 08:57:33,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25557.00 MB 2025-02-15 08:57:33,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4240.44 MB 2025-02-15 08:57:33,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54588.87 MB 2025-02-15 08:57:33,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 08:57:33,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25528.63 MB 2025-02-15 08:57:33,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34411.81 MB 2025-02-15 08:57:33,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:57:33,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:57:33,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:57:33,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:33,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25557.00 MB 2025-02-15 08:57:33,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22005.88 MB 2025-02-15 08:57:33,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3551.12 MB 2025-02-15 08:57:33,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 08:57:33,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 08:57:33,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6994.00 MB 2025-02-15 08:57:33,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35358.46 MB 2025-02-15 08:57:35,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:57:35,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:57:35,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 08:57:35,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:35,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22005.88 MB 2025-02-15 08:57:35,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22536.72 MB 2025-02-15 08:57:35,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:57:35,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36054.24 MB 2025-02-15 08:57:35,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26235.37 MB 2025-02-15 08:57:35,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9818.87 MB 2025-02-15 08:57:35,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26516.31 MB 2025-02-15 08:57:35,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:57:35,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:57:35,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:57:35,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:35,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22536.72 MB 2025-02-15 08:57:35,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24426.25 MB 2025-02-15 08:57:35,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:57:35,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26235.37 MB 2025-02-15 08:57:35,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28122.81 MB 2025-02-15 08:57:35,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 08:57:35,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25843.68 MB 2025-02-15 08:57:35,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:57:35,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:57:35,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:57:35,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:35,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.25 MB 2025-02-15 08:57:35,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26669.16 MB 2025-02-15 08:57:35,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 08:57:35,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28122.81 MB 2025-02-15 08:57:35,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-15 08:57:35,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 08:57:35,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32213.44 MB 2025-02-15 08:57:35,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:57:35,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:57:35,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:57:35,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:35,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22536.72 MB 2025-02-15 08:57:35,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26669.16 MB 2025-02-15 08:57:35,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 08:57:35,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26235.37 MB 2025-02-15 08:57:35,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-15 08:57:35,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8258.58 MB 2025-02-15 08:57:35,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32213.44 MB 2025-02-15 08:57:36,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:57:36,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:57:36,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 08:57:36,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:36,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28202.70 MB 2025-02-15 08:57:36,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28969.70 MB 2025-02-15 08:57:36,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:57:36,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-15 08:57:36,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 08:57:36,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:57:36,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29677.49 MB 2025-02-15 08:57:36,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:57:36,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:57:36,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:57:36,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:36,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29382.59 MB 2025-02-15 08:57:36,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29611.45 MB 2025-02-15 08:57:36,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-15 08:57:36,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 08:57:36,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 08:57:36,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:57:36,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29824.73 MB 2025-02-15 08:57:36,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:57:36,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:57:36,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.05 seconds 2025-02-15 08:57:36,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:36,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17142.63 MB 2025-02-15 08:57:36,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29812.31 MB 2025-02-15 08:57:36,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12669.67 MB 2025-02-15 08:57:36,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54588.87 MB 2025-02-15 08:57:36,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 08:57:36,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19677.58 MB 2025-02-15 08:57:36,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29824.73 MB 2025-02-15 08:57:36,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:57:36,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:57:36,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 08:57:36,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:36,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29812.31 MB 2025-02-15 08:57:36,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22142.53 MB 2025-02-15 08:57:36,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7669.78 MB 2025-02-15 08:57:36,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 08:57:36,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 08:57:36,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:57:36,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32320.29 MB 2025-02-15 08:57:36,390 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 08:57:36,391 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:57:36,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:57:36,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:57:36,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:57:36,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:57:36,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22142.53 MB 2025-02-15 08:57:36,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30569.03 MB 2025-02-15 08:57:36,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 08:57:36,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 08:57:36,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43289.41 MB 2025-02-15 08:57:36,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-15 08:57:36,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30569.03 MB 2025-02-15 08:57:36,647 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 08:57:36,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:36,650 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:57:36,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:36,652 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:57:36,659 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:57:36,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:57:36,661 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:57:36,661 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 08:58:27,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:27,199 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:58:27,204 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:58:27,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:27,207 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1005, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:58:27,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:27,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1005, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:58:42,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:58:42,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:58:42,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.52 seconds 2025-02-15 08:58:42,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:42,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19971.71 MB 2025-02-15 08:58:42,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23528.48 MB 2025-02-15 08:58:42,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3556.77 MB 2025-02-15 08:58:42,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55855.55 MB 2025-02-15 08:58:42,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28376.56 MB 2025-02-15 08:58:42,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27478.98 MB 2025-02-15 08:58:42,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32387.48 MB 2025-02-15 08:58:42,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:58:42,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:58:42,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 08:58:42,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:42,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23528.48 MB 2025-02-15 08:58:42,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21003.58 MB 2025-02-15 08:58:42,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2524.90 MB 2025-02-15 08:58:42,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28376.56 MB 2025-02-15 08:58:42,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37559.99 MB 2025-02-15 08:58:42,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9183.43 MB 2025-02-15 08:58:42,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34813.32 MB 2025-02-15 08:58:44,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:58:44,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:58:44,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 08:58:44,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:44,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21003.58 MB 2025-02-15 08:58:44,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21534.42 MB 2025-02-15 08:58:44,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:58:44,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37559.99 MB 2025-02-15 08:58:44,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26944.21 MB 2025-02-15 08:58:44,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10615.78 MB 2025-02-15 08:58:44,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25512.97 MB 2025-02-15 08:58:44,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:58:44,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:58:44,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:58:44,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:44,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21534.42 MB 2025-02-15 08:58:44,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23423.96 MB 2025-02-15 08:58:44,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:58:44,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 08:58:44,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26944.21 MB 2025-02-15 08:58:44,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:58:44,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24841.38 MB 2025-02-15 08:58:44,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:58:44,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:58:44,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:58:44,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:44,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23423.96 MB 2025-02-15 08:58:44,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25665.81 MB 2025-02-15 08:58:44,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:58:44,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 08:58:44,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 08:58:44,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:58:44,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.09 MB 2025-02-15 08:58:44,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:58:44,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:58:44,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:58:44,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:44,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21534.42 MB 2025-02-15 08:58:44,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25665.81 MB 2025-02-15 08:58:44,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:58:44,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 08:58:44,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 08:58:44,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:58:44,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.09 MB 2025-02-15 08:58:45,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:58:45,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:58:45,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:58:45,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:45,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27199.35 MB 2025-02-15 08:58:45,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27966.36 MB 2025-02-15 08:58:45,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:58:45,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33550.24 MB 2025-02-15 08:58:45,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:58:45,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 08:58:45,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28674.14 MB 2025-02-15 08:58:45,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:58:45,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:58:45,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:58:45,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:45,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28379.25 MB 2025-02-15 08:58:45,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28606.60 MB 2025-02-15 08:58:45,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.35 MB 2025-02-15 08:58:45,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:58:45,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:58:45,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:58:45,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28841.70 MB 2025-02-15 08:58:45,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:58:45,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:58:45,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.92 seconds 2025-02-15 08:58:45,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:45,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16470.21 MB 2025-02-15 08:58:45,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28806.56 MB 2025-02-15 08:58:45,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12336.36 MB 2025-02-15 08:58:45,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55855.55 MB 2025-02-15 08:58:45,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:58:45,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21887.98 MB 2025-02-15 08:58:45,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28841.70 MB 2025-02-15 08:58:45,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:58:45,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:58:45,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:58:45,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:45,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28806.56 MB 2025-02-15 08:58:45,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21457.45 MB 2025-02-15 08:58:45,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7349.11 MB 2025-02-15 08:58:45,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:58:45,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:58:45,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:58:45,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31304.41 MB 2025-02-15 08:58:45,418 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 08:58:45,419 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 08:58:45,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:58:45,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:58:45,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:58:45,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:58:45,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21457.45 MB 2025-02-15 08:58:45,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29850.05 MB 2025-02-15 08:58:45,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-15 08:58:45,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:58:45,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42312.14 MB 2025-02-15 08:58:45,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-15 08:58:45,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29850.05 MB 2025-02-15 08:58:45,582 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 08:58:45,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:45,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:58:45,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:45,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:58:45,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:58:45,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:58:45,590 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:58:45,590 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 08:59:08,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:08,596 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 08:59:08,600 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 08:59:08,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:08,604 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 08:59:08,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:08,605 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 08:59:25,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 08:59:25,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 08:59:25,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.67 seconds 2025-02-15 08:59:25,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:25,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-15 08:59:25,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-15 08:59:25,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-15 08:59:25,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54827.94 MB 2025-02-15 08:59:25,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28624.03 MB 2025-02-15 08:59:25,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26203.91 MB 2025-02-15 08:59:25,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33094.78 MB 2025-02-15 08:59:25,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 08:59:25,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 08:59:25,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 08:59:25,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:25,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-15 08:59:25,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21362.29 MB 2025-02-15 08:59:25,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2891.05 MB 2025-02-15 08:59:25,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28624.03 MB 2025-02-15 08:59:25,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38182.85 MB 2025-02-15 08:59:25,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9558.82 MB 2025-02-15 08:59:25,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35910.19 MB 2025-02-15 08:59:27,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 08:59:27,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 08:59:27,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 08:59:27,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.29 MB 2025-02-15 08:59:27,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21893.13 MB 2025-02-15 08:59:27,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 08:59:27,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38182.85 MB 2025-02-15 08:59:27,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26946.31 MB 2025-02-15 08:59:27,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11236.54 MB 2025-02-15 08:59:27,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25871.68 MB 2025-02-15 08:59:27,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 08:59:27,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 08:59:27,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 08:59:27,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-15 08:59:27,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23782.67 MB 2025-02-15 08:59:27,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 08:59:27,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26946.31 MB 2025-02-15 08:59:27,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-15 08:59:27,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 08:59:27,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25200.10 MB 2025-02-15 08:59:27,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 08:59:27,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 08:59:27,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 08:59:27,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23782.67 MB 2025-02-15 08:59:27,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-15 08:59:27,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 08:59:27,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-15 08:59:27,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33552.33 MB 2025-02-15 08:59:27,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 08:59:27,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-15 08:59:27,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 08:59:27,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 08:59:27,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 08:59:27,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-15 08:59:27,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-15 08:59:27,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 08:59:27,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26946.31 MB 2025-02-15 08:59:27,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33552.33 MB 2025-02-15 08:59:27,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 08:59:27,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-15 08:59:27,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 08:59:27,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 08:59:27,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 08:59:27,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27558.06 MB 2025-02-15 08:59:27,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28325.07 MB 2025-02-15 08:59:27,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 08:59:27,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33552.33 MB 2025-02-15 08:59:27,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:59:27,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 08:59:27,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29032.86 MB 2025-02-15 08:59:27,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 08:59:27,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 08:59:27,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:59:27,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28737.96 MB 2025-02-15 08:59:27,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28965.91 MB 2025-02-15 08:59:27,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-15 08:59:27,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:59:27,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:59:27,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:59:27,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29169.94 MB 2025-02-15 08:59:27,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 08:59:27,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 08:59:27,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.10 seconds 2025-02-15 08:59:27,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-15 08:59:27,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29165.78 MB 2025-02-15 08:59:27,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12455.17 MB 2025-02-15 08:59:27,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54827.94 MB 2025-02-15 08:59:27,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:59:27,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20860.37 MB 2025-02-15 08:59:27,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29169.94 MB 2025-02-15 08:59:27,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 08:59:27,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 08:59:27,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 08:59:27,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29165.78 MB 2025-02-15 08:59:27,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21697.31 MB 2025-02-15 08:59:27,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7468.46 MB 2025-02-15 08:59:27,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:59:27,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 08:59:27,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 08:59:27,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31663.37 MB 2025-02-15 08:59:27,989 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 08:59:27,989 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 08:59:27,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 08:59:27,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 08:59:27,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 08:59:27,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 08:59:27,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21697.31 MB 2025-02-15 08:59:27,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30085.73 MB 2025-02-15 08:59:27,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 08:59:27,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 08:59:27,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42307.94 MB 2025-02-15 08:59:27,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-15 08:59:27,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30085.73 MB 2025-02-15 08:59:28,153 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 08:59:28,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:28,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 08:59:28,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:28,155 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 08:59:28,160 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 08:59:28,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 08:59:28,161 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 08:59:28,161 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:00:24,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:24,849 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:00:24,854 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:00:24,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:24,858 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 506, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:00:24,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:24,859 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 506, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:00:32,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:00:32,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:00:32,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.85 seconds 2025-02-15 09:00:32,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:32,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16494.60 MB 2025-02-15 09:00:32,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18285.30 MB 2025-02-15 09:00:32,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.71 MB 2025-02-15 09:00:32,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54817.46 MB 2025-02-15 09:00:32,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20252.20 MB 2025-02-15 09:00:32,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34565.26 MB 2025-02-15 09:00:32,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27098.43 MB 2025-02-15 09:00:32,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:00:32,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:00:32,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:00:32,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:32,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18285.30 MB 2025-02-15 09:00:32,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18409.44 MB 2025-02-15 09:00:32,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 124.13 MB 2025-02-15 09:00:32,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20252.20 MB 2025-02-15 09:00:32,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25870.47 MB 2025-02-15 09:00:32,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5618.27 MB 2025-02-15 09:00:32,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26002.23 MB 2025-02-15 09:00:34,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:00:34,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:00:34,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 09:00:34,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:34,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18409.44 MB 2025-02-15 09:00:34,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18940.28 MB 2025-02-15 09:00:34,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:00:34,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25870.47 MB 2025-02-15 09:00:34,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 09:00:34,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4437.57 MB 2025-02-15 09:00:34,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22920.90 MB 2025-02-15 09:00:34,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:00:34,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:00:34,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:00:34,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:34,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18940.28 MB 2025-02-15 09:00:34,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20829.81 MB 2025-02-15 09:00:34,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:00:34,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 09:00:34,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24264.05 MB 2025-02-15 09:00:34,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:00:34,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22247.24 MB 2025-02-15 09:00:34,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:00:34,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:00:34,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:00:34,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:34,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20829.81 MB 2025-02-15 09:00:34,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23071.67 MB 2025-02-15 09:00:34,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:00:34,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24264.05 MB 2025-02-15 09:00:34,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 09:00:34,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:00:34,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.95 MB 2025-02-15 09:00:34,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:00:34,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:00:34,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:00:34,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:34,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18940.28 MB 2025-02-15 09:00:34,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23071.67 MB 2025-02-15 09:00:34,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:00:34,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 09:00:34,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 09:00:34,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:00:34,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.95 MB 2025-02-15 09:00:35,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:00:35,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:00:35,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:00:35,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:35,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24605.21 MB 2025-02-15 09:00:35,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25372.21 MB 2025-02-15 09:00:35,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:00:35,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30398.22 MB 2025-02-15 09:00:35,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:00:35,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:00:35,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26080.00 MB 2025-02-15 09:00:35,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:00:35,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:00:35,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:00:35,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:35,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25785.10 MB 2025-02-15 09:00:35,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26012.66 MB 2025-02-15 09:00:35,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.56 MB 2025-02-15 09:00:35,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:00:35,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:00:35,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:00:35,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26238.41 MB 2025-02-15 09:00:35,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:00:35,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:00:35,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.30 seconds 2025-02-15 09:00:35,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:35,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14731.65 MB 2025-02-15 09:00:35,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26213.24 MB 2025-02-15 09:00:35,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11481.59 MB 2025-02-15 09:00:35,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54817.46 MB 2025-02-15 09:00:35,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:00:35,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24001.90 MB 2025-02-15 09:00:35,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26238.41 MB 2025-02-15 09:00:35,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:00:35,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:00:35,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:00:35,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:35,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26213.24 MB 2025-02-15 09:00:35,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19728.69 MB 2025-02-15 09:00:35,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6484.55 MB 2025-02-15 09:00:35,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:00:35,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:00:35,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:00:35,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28718.76 MB 2025-02-15 09:00:35,452 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 09:00:35,453 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:00:35,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:00:35,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:00:35,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:00:35,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:00:35,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19728.69 MB 2025-02-15 09:00:35,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28146.85 MB 2025-02-15 09:00:35,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 09:00:35,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:00:35,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41278.24 MB 2025-02-15 09:00:35,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-15 09:00:35,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28146.85 MB 2025-02-15 09:00:35,617 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 09:00:35,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:35,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:00:35,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:35,619 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:00:35,624 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:00:35,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:35,625 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:00:35,625 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:00:59,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:59,152 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:00:59,157 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:00:59,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:59,160 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:00:59,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:00:59,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:01:20,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:01:20,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:01:20,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.84 seconds 2025-02-15 09:01:20,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:20,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22340.88 MB 2025-02-15 09:01:20,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27101.42 MB 2025-02-15 09:01:20,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4760.54 MB 2025-02-15 09:01:20,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53831.79 MB 2025-02-15 09:01:20,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37977.33 MB 2025-02-15 09:01:20,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15854.47 MB 2025-02-15 09:01:20,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36115.61 MB 2025-02-15 09:01:20,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:01:20,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:01:20,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 09:01:20,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:20,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27101.42 MB 2025-02-15 09:01:20,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22770.09 MB 2025-02-15 09:01:20,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4331.33 MB 2025-02-15 09:01:20,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37977.33 MB 2025-02-15 09:01:20,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47089.45 MB 2025-02-15 09:01:20,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9112.13 MB 2025-02-15 09:01:20,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40801.73 MB 2025-02-15 09:01:22,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:01:22,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:01:22,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:01:22,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22770.09 MB 2025-02-15 09:01:22,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23300.93 MB 2025-02-15 09:01:22,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:01:22,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47089.45 MB 2025-02-15 09:01:22,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29032.97 MB 2025-02-15 09:01:22,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18056.48 MB 2025-02-15 09:01:22,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27279.47 MB 2025-02-15 09:01:22,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:01:22,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:01:22,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:01:22,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23300.93 MB 2025-02-15 09:01:22,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25190.46 MB 2025-02-15 09:01:22,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:01:22,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29032.97 MB 2025-02-15 09:01:22,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29976.69 MB 2025-02-15 09:01:22,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:01:22,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26607.89 MB 2025-02-15 09:01:22,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:01:22,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:01:22,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:01:22,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.46 MB 2025-02-15 09:01:22,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27432.32 MB 2025-02-15 09:01:22,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:01:22,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29976.69 MB 2025-02-15 09:01:22,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35639.00 MB 2025-02-15 09:01:22,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:01:22,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32976.60 MB 2025-02-15 09:01:22,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:01:22,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:01:22,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:01:22,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23300.93 MB 2025-02-15 09:01:22,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27432.32 MB 2025-02-15 09:01:22,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:01:22,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29032.97 MB 2025-02-15 09:01:22,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35639.00 MB 2025-02-15 09:01:22,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:01:22,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32976.60 MB 2025-02-15 09:01:22,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:01:22,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:01:22,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:01:22,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28965.86 MB 2025-02-15 09:01:22,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29732.86 MB 2025-02-15 09:01:22,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:01:22,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35639.00 MB 2025-02-15 09:01:22,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 09:01:22,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:01:22,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30440.65 MB 2025-02-15 09:01:22,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:01:22,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:01:22,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:01:22,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30145.75 MB 2025-02-15 09:01:22,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30374.71 MB 2025-02-15 09:01:22,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 09:01:22,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36054.24 MB 2025-02-15 09:01:22,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 09:01:22,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:01:22,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30594.15 MB 2025-02-15 09:01:22,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:01:22,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:01:22,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.25 seconds 2025-02-15 09:01:22,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17654.79 MB 2025-02-15 09:01:22,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30574.68 MB 2025-02-15 09:01:22,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12919.88 MB 2025-02-15 09:01:22,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53831.79 MB 2025-02-15 09:01:22,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 09:01:22,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17777.56 MB 2025-02-15 09:01:22,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30594.15 MB 2025-02-15 09:01:22,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:01:22,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:01:22,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:01:22,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30574.68 MB 2025-02-15 09:01:22,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22642.93 MB 2025-02-15 09:01:22,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7931.75 MB 2025-02-15 09:01:22,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36054.24 MB 2025-02-15 09:01:22,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 09:01:22,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:01:22,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33073.41 MB 2025-02-15 09:01:22,701 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 09:01:22,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:01:22,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:01:22,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:01:22,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:01:22,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:01:22,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22642.93 MB 2025-02-15 09:01:22,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31035.52 MB 2025-02-15 09:01:22,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-15 09:01:22,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36054.24 MB 2025-02-15 09:01:22,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44398.80 MB 2025-02-15 09:01:22,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-15 09:01:22,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31035.52 MB 2025-02-15 09:01:22,864 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 09:01:22,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:01:22,866 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:01:22,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:01:22,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:01:22,871 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:01:22,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:01:22,872 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:01:22,872 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:02:30,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:30,696 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:02:30,701 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:02:30,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:30,705 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 451, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:02:30,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:30,706 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 451, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:02:37,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:02:37,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:02:37,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.00 seconds 2025-02-15 09:02:37,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:37,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16111.35 MB 2025-02-15 09:02:37,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17707.41 MB 2025-02-15 09:02:37,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1596.06 MB 2025-02-15 09:02:37,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56914.61 MB 2025-02-15 09:02:37,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20254.29 MB 2025-02-15 09:02:37,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36660.31 MB 2025-02-15 09:02:37,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26715.18 MB 2025-02-15 09:02:37,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:02:37,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:02:37,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:02:37,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:37,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17707.41 MB 2025-02-15 09:02:37,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18123.51 MB 2025-02-15 09:02:37,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 416.10 MB 2025-02-15 09:02:37,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20254.29 MB 2025-02-15 09:02:37,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-15 09:02:37,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5280.63 MB 2025-02-15 09:02:37,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25087.57 MB 2025-02-15 09:02:39,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:02:39,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:02:39,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:02:39,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:39,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18123.51 MB 2025-02-15 09:02:39,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18654.35 MB 2025-02-15 09:02:39,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:02:39,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25534.92 MB 2025-02-15 09:02:39,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21434.99 MB 2025-02-15 09:02:39,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4099.93 MB 2025-02-15 09:02:39,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22633.93 MB 2025-02-15 09:02:39,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:02:39,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:02:39,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:02:39,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:39,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18654.35 MB 2025-02-15 09:02:39,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20543.88 MB 2025-02-15 09:02:39,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:02:39,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 09:02:39,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24266.15 MB 2025-02-15 09:02:39,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:02:39,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21961.31 MB 2025-02-15 09:02:39,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:02:39,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:02:39,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:02:39,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:39,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20543.88 MB 2025-02-15 09:02:39,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22785.74 MB 2025-02-15 09:02:39,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:02:39,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24266.15 MB 2025-02-15 09:02:39,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 09:02:39,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:02:39,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28330.02 MB 2025-02-15 09:02:39,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:02:39,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:02:39,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:02:39,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:39,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18654.35 MB 2025-02-15 09:02:39,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22785.74 MB 2025-02-15 09:02:39,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:02:39,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 09:02:39,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 09:02:39,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:02:39,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28330.02 MB 2025-02-15 09:02:40,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:02:40,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:02:40,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:02:40,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:40,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24319.28 MB 2025-02-15 09:02:40,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25086.28 MB 2025-02-15 09:02:40,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:02:40,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-15 09:02:40,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:02:40,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:02:40,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25794.07 MB 2025-02-15 09:02:40,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:02:40,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:02:40,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:02:40,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:40,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25499.17 MB 2025-02-15 09:02:40,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25728.47 MB 2025-02-15 09:02:40,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.29 MB 2025-02-15 09:02:40,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:02:40,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:02:40,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:02:40,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25941.94 MB 2025-02-15 09:02:40,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:02:40,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:02:40,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.40 seconds 2025-02-15 09:02:40,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:40,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.03 MB 2025-02-15 09:02:40,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25929.54 MB 2025-02-15 09:02:40,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11389.51 MB 2025-02-15 09:02:40,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56914.61 MB 2025-02-15 09:02:40,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:02:40,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26099.06 MB 2025-02-15 09:02:40,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25941.94 MB 2025-02-15 09:02:40,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:02:40,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:02:40,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:02:40,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:40,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.54 MB 2025-02-15 09:02:40,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19544.42 MB 2025-02-15 09:02:40,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6385.12 MB 2025-02-15 09:02:40,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:02:40,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:02:40,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:02:40,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28441.21 MB 2025-02-15 09:02:40,391 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:02:40,391 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:02:40,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:02:40,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:02:40,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:02:40,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:02:40,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19544.42 MB 2025-02-15 09:02:40,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27983.44 MB 2025-02-15 09:02:40,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:02:40,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:02:40,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 09:02:40,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 09:02:40,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27983.44 MB 2025-02-15 09:02:40,562 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:02:40,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:40,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:02:40,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:40,564 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:02:40,569 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:02:40,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:02:40,570 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:02:40,570 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:04:49,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:04:49,624 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:04:49,629 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:04:49,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:04:49,633 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1374, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:04:49,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:04:49,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1374, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:05:10,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:05:10,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:05:10,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.96 seconds 2025-02-15 09:05:10,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:10,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.96 MB 2025-02-15 09:05:10,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.25 MB 2025-02-15 09:05:10,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4863.30 MB 2025-02-15 09:05:10,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 09:05:10,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38122.03 MB 2025-02-15 09:05:10,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15768.49 MB 2025-02-15 09:05:10,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36317.69 MB 2025-02-15 09:05:10,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:05:10,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:05:10,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 09:05:10,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:10,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27406.25 MB 2025-02-15 09:05:10,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22920.85 MB 2025-02-15 09:05:10,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4485.41 MB 2025-02-15 09:05:10,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38122.03 MB 2025-02-15 09:05:10,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44365.25 MB 2025-02-15 09:05:10,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6243.22 MB 2025-02-15 09:05:10,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39974.92 MB 2025-02-15 09:05:12,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:05:12,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:05:12,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 09:05:12,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:12,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22920.85 MB 2025-02-15 09:05:12,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23451.69 MB 2025-02-15 09:05:12,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:05:12,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44365.25 MB 2025-02-15 09:05:12,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 09:05:12,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15300.82 MB 2025-02-15 09:05:12,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27430.24 MB 2025-02-15 09:05:12,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:05:12,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:05:12,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:05:12,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:12,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23451.69 MB 2025-02-15 09:05:12,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25341.22 MB 2025-02-15 09:05:12,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:05:12,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 09:05:12,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30008.15 MB 2025-02-15 09:05:12,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:05:12,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26758.65 MB 2025-02-15 09:05:12,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:05:12,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:05:12,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:05:12,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:12,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25341.22 MB 2025-02-15 09:05:12,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.08 MB 2025-02-15 09:05:12,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:05:12,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30008.15 MB 2025-02-15 09:05:12,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35670.46 MB 2025-02-15 09:05:12,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:05:12,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33127.36 MB 2025-02-15 09:05:12,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:05:12,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:05:12,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:05:12,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:12,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23451.69 MB 2025-02-15 09:05:12,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.08 MB 2025-02-15 09:05:12,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:05:12,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 09:05:12,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35670.46 MB 2025-02-15 09:05:12,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:05:12,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33127.36 MB 2025-02-15 09:05:12,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:05:12,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:05:12,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:05:12,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:12,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29116.62 MB 2025-02-15 09:05:12,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29883.62 MB 2025-02-15 09:05:12,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:05:12,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35670.46 MB 2025-02-15 09:05:12,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 09:05:12,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:05:12,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30591.41 MB 2025-02-15 09:05:13,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:05:13,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:05:13,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:05:13,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:13,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30296.51 MB 2025-02-15 09:05:13,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30524.88 MB 2025-02-15 09:05:13,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 09:05:13,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 09:05:13,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 09:05:13,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:05:13,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30739.15 MB 2025-02-15 09:05:13,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:05:13,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:05:13,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.37 seconds 2025-02-15 09:05:13,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:13,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17755.83 MB 2025-02-15 09:05:13,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30725.74 MB 2025-02-15 09:05:13,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12969.90 MB 2025-02-15 09:05:13,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 09:05:13,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 09:05:13,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17804.82 MB 2025-02-15 09:05:13,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30739.15 MB 2025-02-15 09:05:13,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:05:13,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:05:13,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:05:13,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:13,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30725.74 MB 2025-02-15 09:05:13,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22748.60 MB 2025-02-15 09:05:13,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7977.14 MB 2025-02-15 09:05:13,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 09:05:13,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 09:05:13,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:05:13,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33227.57 MB 2025-02-15 09:05:13,295 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 09:05:13,295 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:05:13,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:05:13,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:05:13,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 09:05:13,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:05:13,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22748.60 MB 2025-02-15 09:05:13,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31154.26 MB 2025-02-15 09:05:13,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 09:05:13,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 09:05:13,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44444.94 MB 2025-02-15 09:05:13,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 09:05:13,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31154.26 MB 2025-02-15 09:05:13,468 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 09:05:13,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:05:13,469 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:05:13,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:05:13,470 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:05:13,475 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:05:13,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:05:13,476 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:05:13,476 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:06:48,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:06:48,238 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:06:48,243 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:06:48,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:06:48,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2771, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:06:48,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:06:48,248 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2771, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:07:31,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:07:31,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:07:31,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.82 seconds 2025-02-15 09:07:31,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:31,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32279.41 MB 2025-02-15 09:07:31,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42085.83 MB 2025-02-15 09:07:31,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9806.41 MB 2025-02-15 09:07:31,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72118.96 MB 2025-02-15 09:07:31,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45604.67 MB 2025-02-15 09:07:31,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26514.29 MB 2025-02-15 09:07:31,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51892.24 MB 2025-02-15 09:07:31,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:07:31,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:07:31,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 09:07:31,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:31,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42085.83 MB 2025-02-15 09:07:31,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30186.40 MB 2025-02-15 09:07:31,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11899.43 MB 2025-02-15 09:07:31,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45604.67 MB 2025-02-15 09:07:31,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66246.93 MB 2025-02-15 09:07:31,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20642.27 MB 2025-02-15 09:07:31,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69930.43 MB 2025-02-15 09:07:33,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:07:33,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:07:33,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:07:33,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30186.40 MB 2025-02-15 09:07:33,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30717.24 MB 2025-02-15 09:07:33,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:07:33,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66246.93 MB 2025-02-15 09:07:33,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33015.46 MB 2025-02-15 09:07:33,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33231.47 MB 2025-02-15 09:07:33,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34695.79 MB 2025-02-15 09:07:33,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:07:33,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:07:33,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:07:33,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30717.24 MB 2025-02-15 09:07:33,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32606.71 MB 2025-02-15 09:07:33,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-15 09:07:33,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33015.46 MB 2025-02-15 09:07:33,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35846.62 MB 2025-02-15 09:07:33,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:07:33,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34024.14 MB 2025-02-15 09:07:33,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:07:33,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:07:33,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:07:33,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32606.71 MB 2025-02-15 09:07:33,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34848.56 MB 2025-02-15 09:07:33,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:07:33,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35846.62 MB 2025-02-15 09:07:33,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41980.79 MB 2025-02-15 09:07:33,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:07:33,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40392.85 MB 2025-02-15 09:07:33,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:07:33,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:07:33,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:07:33,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30717.24 MB 2025-02-15 09:07:33,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34848.56 MB 2025-02-15 09:07:33,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-15 09:07:33,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33015.46 MB 2025-02-15 09:07:33,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41980.79 MB 2025-02-15 09:07:33,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:07:33,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40392.85 MB 2025-02-15 09:07:33,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:07:33,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:07:33,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:07:33,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36382.11 MB 2025-02-15 09:07:33,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37149.11 MB 2025-02-15 09:07:33,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:07:33,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41980.79 MB 2025-02-15 09:07:33,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42393.93 MB 2025-02-15 09:07:33,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:07:33,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37856.90 MB 2025-02-15 09:07:33,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:07:33,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:07:33,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:07:33,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37562.00 MB 2025-02-15 09:07:33,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37790.93 MB 2025-02-15 09:07:33,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-15 09:07:33,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42393.93 MB 2025-02-15 09:07:33,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42393.93 MB 2025-02-15 09:07:33,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:07:33,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38015.30 MB 2025-02-15 09:07:33,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:07:33,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:07:33,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.48 seconds 2025-02-15 09:07:33,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22624.06 MB 2025-02-15 09:07:33,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37991.79 MB 2025-02-15 09:07:33,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15367.73 MB 2025-02-15 09:07:33,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62461.58 MB 2025-02-15 09:07:33,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42393.93 MB 2025-02-15 09:07:33,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20067.65 MB 2025-02-15 09:07:33,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38015.30 MB 2025-02-15 09:07:33,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:07:33,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:07:33,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:07:33,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:33,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37991.79 MB 2025-02-15 09:07:33,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27625.02 MB 2025-02-15 09:07:33,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10366.77 MB 2025-02-15 09:07:33,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42393.93 MB 2025-02-15 09:07:33,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42393.93 MB 2025-02-15 09:07:33,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:07:33,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40500.69 MB 2025-02-15 09:07:34,015 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 09:07:34,015 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:07:34,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:07:34,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:07:34,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:07:34,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:07:34,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27625.02 MB 2025-02-15 09:07:34,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36054.66 MB 2025-02-15 09:07:34,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.64 MB 2025-02-15 09:07:34,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42393.93 MB 2025-02-15 09:07:34,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46584.04 MB 2025-02-15 09:07:34,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 09:07:34,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36054.66 MB 2025-02-15 09:07:34,179 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 09:07:34,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:07:34,181 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:07:34,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:07:34,182 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:07:34,186 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:07:34,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:07:34,187 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:07:34,188 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:08:35,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:08:35,060 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:08:35,065 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:08:35,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:08:35,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1978, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:08:35,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:08:35,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1978, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:09:05,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:09:05,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:09:05,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.67 seconds 2025-02-15 09:09:05,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:05,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26751.73 MB 2025-02-15 09:09:05,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33752.02 MB 2025-02-15 09:09:05,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7000.29 MB 2025-02-15 09:09:05,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54964.26 MB 2025-02-15 09:09:05,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40802.19 MB 2025-02-15 09:09:05,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14162.07 MB 2025-02-15 09:09:05,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42564.88 MB 2025-02-15 09:09:05,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:09:05,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:09:05,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:09:05,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:05,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33752.02 MB 2025-02-15 09:09:05,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26060.86 MB 2025-02-15 09:09:05,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7691.16 MB 2025-02-15 09:09:05,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40802.19 MB 2025-02-15 09:09:05,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55710.84 MB 2025-02-15 09:09:05,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14908.65 MB 2025-02-15 09:09:05,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54045.02 MB 2025-02-15 09:09:07,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:09:07,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:09:07,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:09:07,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:07,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26060.86 MB 2025-02-15 09:09:07,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26591.70 MB 2025-02-15 09:09:07,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:09:07,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55710.84 MB 2025-02-15 09:09:07,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35217.47 MB 2025-02-15 09:09:07,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20493.37 MB 2025-02-15 09:09:07,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.24 MB 2025-02-15 09:09:07,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:09:07,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:09:07,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:09:07,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:07,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.70 MB 2025-02-15 09:09:07,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28481.23 MB 2025-02-15 09:09:07,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:09:07,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35217.47 MB 2025-02-15 09:09:07,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35217.47 MB 2025-02-15 09:09:07,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:09:07,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29898.66 MB 2025-02-15 09:09:08,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:09:08,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:09:08,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:09:08,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28481.23 MB 2025-02-15 09:09:08,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30723.09 MB 2025-02-15 09:09:08,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:09:08,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35217.47 MB 2025-02-15 09:09:08,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38520.49 MB 2025-02-15 09:09:08,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 09:09:08,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36267.37 MB 2025-02-15 09:09:08,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:09:08,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:09:08,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:09:08,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.70 MB 2025-02-15 09:09:08,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30723.09 MB 2025-02-15 09:09:08,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:09:08,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35217.47 MB 2025-02-15 09:09:08,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38520.49 MB 2025-02-15 09:09:08,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 09:09:08,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36267.37 MB 2025-02-15 09:09:08,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:09:08,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:09:08,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:09:08,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32256.63 MB 2025-02-15 09:09:08,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33023.63 MB 2025-02-15 09:09:08,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:09:08,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38520.49 MB 2025-02-15 09:09:08,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38935.72 MB 2025-02-15 09:09:08,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:09:08,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33731.42 MB 2025-02-15 09:09:08,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:09:08,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:09:08,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:09:08,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33436.52 MB 2025-02-15 09:09:08,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33665.60 MB 2025-02-15 09:09:08,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-15 09:09:08,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38935.72 MB 2025-02-15 09:09:08,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38935.72 MB 2025-02-15 09:09:08,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:09:08,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33884.61 MB 2025-02-15 09:09:08,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:09:08,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:09:08,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.18 seconds 2025-02-15 09:09:08,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19860.22 MB 2025-02-15 09:09:08,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33866.60 MB 2025-02-15 09:09:08,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14006.39 MB 2025-02-15 09:09:08,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54964.26 MB 2025-02-15 09:09:08,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38935.72 MB 2025-02-15 09:09:08,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16028.53 MB 2025-02-15 09:09:08,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33884.61 MB 2025-02-15 09:09:08,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:09:08,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:09:08,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:09:08,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33866.60 MB 2025-02-15 09:09:08,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24863.46 MB 2025-02-15 09:09:08,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9003.14 MB 2025-02-15 09:09:08,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38935.72 MB 2025-02-15 09:09:08,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38935.72 MB 2025-02-15 09:09:08,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:09:08,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36377.35 MB 2025-02-15 09:09:08,543 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 09:09:08,543 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:09:08,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:09:08,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:09:08,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:09:08,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:09:08,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24863.46 MB 2025-02-15 09:09:08,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33299.06 MB 2025-02-15 09:09:08,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 09:09:08,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38935.72 MB 2025-02-15 09:09:08,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43130.03 MB 2025-02-15 09:09:08,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 09:09:08,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33299.06 MB 2025-02-15 09:09:08,714 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 09:09:08,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:08,715 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:09:08,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:08,716 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:09:08,721 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:09:08,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:08,722 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:09:08,722 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:09:42,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:42,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:09:42,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:09:42,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:42,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1294, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:09:42,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:09:42,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1294, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:10:02,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:10:02,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:10:02,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.15 seconds 2025-02-15 09:10:02,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:02,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21985.51 MB 2025-02-15 09:10:02,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26565.69 MB 2025-02-15 09:10:02,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4580.18 MB 2025-02-15 09:10:02,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51518.64 MB 2025-02-15 09:10:02,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37824.23 MB 2025-02-15 09:10:02,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13694.40 MB 2025-02-15 09:10:02,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35533.74 MB 2025-02-15 09:10:02,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:10:02,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:10:02,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 09:10:02,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:02,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.69 MB 2025-02-15 09:10:02,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22504.95 MB 2025-02-15 09:10:02,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4060.73 MB 2025-02-15 09:10:02,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37824.23 MB 2025-02-15 09:10:02,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46569.36 MB 2025-02-15 09:10:02,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8745.12 MB 2025-02-15 09:10:02,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39715.72 MB 2025-02-15 09:10:04,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:10:04,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:10:04,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:10:04,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22504.95 MB 2025-02-15 09:10:04,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23035.79 MB 2025-02-15 09:10:04,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:10:04,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46569.36 MB 2025-02-15 09:10:04,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 09:10:04,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13325.30 MB 2025-02-15 09:10:04,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27014.34 MB 2025-02-15 09:10:04,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:10:04,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:10:04,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:10:04,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23035.79 MB 2025-02-15 09:10:04,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24925.33 MB 2025-02-15 09:10:04,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:10:04,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 09:10:04,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33244.05 MB 2025-02-15 09:10:04,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:04,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26342.76 MB 2025-02-15 09:10:04,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:10:04,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:10:04,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:10:04,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24925.33 MB 2025-02-15 09:10:04,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27167.18 MB 2025-02-15 09:10:04,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:10:04,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 09:10:04,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34187.77 MB 2025-02-15 09:10:04,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:10:04,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.47 MB 2025-02-15 09:10:04,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:10:04,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:10:04,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:10:04,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23035.79 MB 2025-02-15 09:10:04,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27167.18 MB 2025-02-15 09:10:04,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:10:04,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33244.05 MB 2025-02-15 09:10:04,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34187.77 MB 2025-02-15 09:10:04,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:10:04,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.47 MB 2025-02-15 09:10:04,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:10:04,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:10:04,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:10:04,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28700.73 MB 2025-02-15 09:10:04,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29467.73 MB 2025-02-15 09:10:04,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:10:04,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34187.77 MB 2025-02-15 09:10:04,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34603.01 MB 2025-02-15 09:10:04,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:10:04,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30175.52 MB 2025-02-15 09:10:04,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:10:04,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:10:04,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:10:04,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29880.62 MB 2025-02-15 09:10:04,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30108.91 MB 2025-02-15 09:10:04,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-15 09:10:04,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34603.01 MB 2025-02-15 09:10:04,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34603.01 MB 2025-02-15 09:10:04,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:04,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30316.68 MB 2025-02-15 09:10:04,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:10:04,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:10:04,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.57 seconds 2025-02-15 09:10:04,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17477.11 MB 2025-02-15 09:10:04,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30309.86 MB 2025-02-15 09:10:04,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12832.76 MB 2025-02-15 09:10:04,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51518.64 MB 2025-02-15 09:10:04,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34603.01 MB 2025-02-15 09:10:04,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16915.63 MB 2025-02-15 09:10:04,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30316.68 MB 2025-02-15 09:10:04,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:10:04,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:10:04,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:10:04,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30309.86 MB 2025-02-15 09:10:04,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22479.59 MB 2025-02-15 09:10:04,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7830.27 MB 2025-02-15 09:10:04,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34603.01 MB 2025-02-15 09:10:04,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34603.01 MB 2025-02-15 09:10:04,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:04,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32820.00 MB 2025-02-15 09:10:04,984 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 09:10:04,984 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:10:04,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:10:04,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:10:04,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:10:04,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:04,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22479.59 MB 2025-02-15 09:10:04,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30914.21 MB 2025-02-15 09:10:04,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 09:10:04,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34603.01 MB 2025-02-15 09:10:04,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42987.42 MB 2025-02-15 09:10:04,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 09:10:04,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30914.21 MB 2025-02-15 09:10:05,151 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 09:10:05,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:05,153 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:10:05,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:05,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:10:05,158 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:10:05,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:05,159 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:10:05,159 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:10:42,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:42,232 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:10:42,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:10:42,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:42,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:10:42,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:42,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:10:53,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:10:53,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:10:53,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.76 seconds 2025-02-15 09:10:53,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:53,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17790.67 MB 2025-02-15 09:10:53,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20240.15 MB 2025-02-15 09:10:53,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2449.47 MB 2025-02-15 09:10:53,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-15 09:10:53,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27315.40 MB 2025-02-15 09:10:53,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24056.43 MB 2025-02-15 09:10:53,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29073.98 MB 2025-02-15 09:10:53,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:10:53,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:10:53,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:10:53,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:53,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20240.15 MB 2025-02-15 09:10:53,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19375.34 MB 2025-02-15 09:10:53,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -864.80 MB 2025-02-15 09:10:53,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27315.40 MB 2025-02-15 09:10:53,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30823.94 MB 2025-02-15 09:10:53,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3508.54 MB 2025-02-15 09:10:53,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.54 MB 2025-02-15 09:10:54,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:10:54,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:10:54,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:10:54,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:54,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19375.34 MB 2025-02-15 09:10:54,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19906.18 MB 2025-02-15 09:10:54,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:10:54,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30823.94 MB 2025-02-15 09:10:54,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26281.51 MB 2025-02-15 09:10:54,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4542.43 MB 2025-02-15 09:10:54,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23884.73 MB 2025-02-15 09:10:54,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:10:54,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:10:54,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:10:54,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:54,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19906.18 MB 2025-02-15 09:10:54,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21795.72 MB 2025-02-15 09:10:54,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:10:54,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26281.51 MB 2025-02-15 09:10:54,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26281.51 MB 2025-02-15 09:10:54,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:54,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23213.15 MB 2025-02-15 09:10:55,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:10:55,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:10:55,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:10:55,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21795.72 MB 2025-02-15 09:10:55,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24037.57 MB 2025-02-15 09:10:55,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:10:55,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26281.51 MB 2025-02-15 09:10:55,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 09:10:55,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:10:55,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29581.86 MB 2025-02-15 09:10:55,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:10:55,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:10:55,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:10:55,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19906.18 MB 2025-02-15 09:10:55,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24037.57 MB 2025-02-15 09:10:55,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:10:55,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26281.51 MB 2025-02-15 09:10:55,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 09:10:55,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:10:55,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29581.86 MB 2025-02-15 09:10:55,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:10:55,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:10:55,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:10:55,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25571.12 MB 2025-02-15 09:10:55,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26338.12 MB 2025-02-15 09:10:55,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:10:55,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31943.82 MB 2025-02-15 09:10:55,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 09:10:55,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:10:55,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27045.91 MB 2025-02-15 09:10:55,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:10:55,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:10:55,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:10:55,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26751.01 MB 2025-02-15 09:10:55,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26980.45 MB 2025-02-15 09:10:55,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.44 MB 2025-02-15 09:10:55,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 09:10:55,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 09:10:55,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:55,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27173.51 MB 2025-02-15 09:10:55,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:10:55,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:10:55,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.15 seconds 2025-02-15 09:10:55,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15379.69 MB 2025-02-15 09:10:55,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27181.52 MB 2025-02-15 09:10:55,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11801.83 MB 2025-02-15 09:10:55,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-15 09:10:55,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 09:10:55,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19014.88 MB 2025-02-15 09:10:55,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27181.52 MB 2025-02-15 09:10:55,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:10:55,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:10:55,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:10:55,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27181.52 MB 2025-02-15 09:10:55,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20384.08 MB 2025-02-15 09:10:55,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6797.44 MB 2025-02-15 09:10:55,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 09:10:55,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 09:10:55,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:10:55,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29693.19 MB 2025-02-15 09:10:55,695 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:10:55,695 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:10:55,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:10:55,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:10:55,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:10:55,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:10:55,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20384.08 MB 2025-02-15 09:10:55,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28823.10 MB 2025-02-15 09:10:55,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:10:55,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 09:10:55,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40747.66 MB 2025-02-15 09:10:55,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:10:55,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28823.10 MB 2025-02-15 09:10:55,862 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:10:55,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:55,864 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:10:55,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:55,865 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:10:55,869 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:10:55,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:10:55,871 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:10:55,871 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:11:06,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:06,865 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:11:06,870 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:11:06,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:06,874 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:11:06,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:06,875 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:11:17,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:11:17,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:11:17,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.39 seconds 2025-02-15 09:11:17,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:17,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.50 MB 2025-02-15 09:11:17,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19966.70 MB 2025-02-15 09:11:17,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2357.20 MB 2025-02-15 09:11:17,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53332.67 MB 2025-02-15 09:11:17,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24442.31 MB 2025-02-15 09:11:17,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28890.37 MB 2025-02-15 09:11:17,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.81 MB 2025-02-15 09:11:17,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:11:17,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:11:17,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:11:17,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:17,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19966.70 MB 2025-02-15 09:11:17,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19241.23 MB 2025-02-15 09:11:17,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -725.47 MB 2025-02-15 09:11:17,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24442.31 MB 2025-02-15 09:11:17,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31092.38 MB 2025-02-15 09:11:17,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6650.07 MB 2025-02-15 09:11:17,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28716.88 MB 2025-02-15 09:11:19,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:11:19,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:11:19,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:11:19,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19241.23 MB 2025-02-15 09:11:19,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19772.07 MB 2025-02-15 09:11:19,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:11:19,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31092.38 MB 2025-02-15 09:11:19,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24209.52 MB 2025-02-15 09:11:19,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6882.85 MB 2025-02-15 09:11:19,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.61 MB 2025-02-15 09:11:19,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:11:19,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:11:19,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:11:19,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19772.07 MB 2025-02-15 09:11:19,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21661.60 MB 2025-02-15 09:11:19,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:11:19,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24209.52 MB 2025-02-15 09:11:19,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26096.96 MB 2025-02-15 09:11:19,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:11:19,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23079.03 MB 2025-02-15 09:11:19,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:11:19,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:11:19,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:11:19,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21661.60 MB 2025-02-15 09:11:19,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23903.46 MB 2025-02-15 09:11:19,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:11:19,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26096.96 MB 2025-02-15 09:11:19,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 09:11:19,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:11:19,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.74 MB 2025-02-15 09:11:19,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:11:19,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:11:19,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:11:19,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19772.07 MB 2025-02-15 09:11:19,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23903.46 MB 2025-02-15 09:11:19,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:11:19,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24209.52 MB 2025-02-15 09:11:19,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 09:11:19,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 09:11:19,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.74 MB 2025-02-15 09:11:19,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:11:19,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:11:19,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:11:19,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25437.00 MB 2025-02-15 09:11:19,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26204.00 MB 2025-02-15 09:11:19,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:11:19,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31759.27 MB 2025-02-15 09:11:19,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32174.51 MB 2025-02-15 09:11:19,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:11:19,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26911.79 MB 2025-02-15 09:11:19,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:11:19,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:11:19,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:11:19,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26616.89 MB 2025-02-15 09:11:19,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26844.68 MB 2025-02-15 09:11:19,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.79 MB 2025-02-15 09:11:19,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32174.51 MB 2025-02-15 09:11:19,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32174.51 MB 2025-02-15 09:11:19,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:11:19,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27077.93 MB 2025-02-15 09:11:19,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:11:19,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:11:19,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.78 seconds 2025-02-15 09:11:19,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15289.10 MB 2025-02-15 09:11:19,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27045.76 MB 2025-02-15 09:11:19,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11756.65 MB 2025-02-15 09:11:19,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53332.67 MB 2025-02-15 09:11:19,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32174.51 MB 2025-02-15 09:11:19,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21158.17 MB 2025-02-15 09:11:19,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27077.93 MB 2025-02-15 09:11:19,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:11:19,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:11:19,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:11:19,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27045.76 MB 2025-02-15 09:11:19,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20293.49 MB 2025-02-15 09:11:19,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6752.26 MB 2025-02-15 09:11:19,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32174.51 MB 2025-02-15 09:11:19,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32174.51 MB 2025-02-15 09:11:19,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:11:19,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29557.42 MB 2025-02-15 09:11:19,945 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:11:19,945 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:11:19,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:11:19,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:11:19,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:11:19,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:11:19,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20293.49 MB 2025-02-15 09:11:19,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28732.52 MB 2025-02-15 09:11:19,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:11:19,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32174.51 MB 2025-02-15 09:11:19,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-15 09:11:19,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:11:19,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28732.52 MB 2025-02-15 09:11:20,109 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:11:20,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:20,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:11:20,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:20,112 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:11:20,116 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:11:20,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:11:20,117 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:11:20,118 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:12:14,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:14,570 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:12:14,575 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:12:14,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:14,578 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:12:14,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:14,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:12:17,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:12:17,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:12:17,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.43 seconds 2025-02-15 09:12:17,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14062.71 MB 2025-02-15 09:12:17,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14618.32 MB 2025-02-15 09:12:17,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 555.61 MB 2025-02-15 09:12:17,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53150.22 MB 2025-02-15 09:12:17,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 09:12:17,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34783.36 MB 2025-02-15 09:12:17,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23534.08 MB 2025-02-15 09:12:17,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:12:17,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:12:17,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:12:17,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14618.32 MB 2025-02-15 09:12:17,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14887.65 MB 2025-02-15 09:12:17,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.32 MB 2025-02-15 09:12:17,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 09:12:17,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18922.60 MB 2025-02-15 09:12:17,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 555.75 MB 2025-02-15 09:12:17,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16823.75 MB 2025-02-15 09:12:17,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:12:17,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:12:17,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.75 seconds 2025-02-15 09:12:17,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14887.65 MB 2025-02-15 09:12:17,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15096.00 MB 2025-02-15 09:12:17,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.36 MB 2025-02-15 09:12:17,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18922.60 MB 2025-02-15 09:12:17,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18922.60 MB 2025-02-15 09:12:17,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:17,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19058.34 MB 2025-02-15 09:12:17,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:12:17,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:12:17,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:12:17,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15095.94 MB 2025-02-15 09:12:17,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15837.40 MB 2025-02-15 09:12:17,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 741.46 MB 2025-02-15 09:12:17,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18922.60 MB 2025-02-15 09:12:17,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18922.60 MB 2025-02-15 09:12:17,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:17,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16393.74 MB 2025-02-15 09:12:17,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:12:17,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:12:17,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:12:17,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15837.40 MB 2025-02-15 09:12:17,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16717.37 MB 2025-02-15 09:12:17,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 879.97 MB 2025-02-15 09:12:17,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18922.60 MB 2025-02-15 09:12:17,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-15 09:12:17,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1855.98 MB 2025-02-15 09:12:17,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18896.60 MB 2025-02-15 09:12:17,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:12:17,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:12:17,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:12:17,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15095.94 MB 2025-02-15 09:12:17,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16717.37 MB 2025-02-15 09:12:17,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1621.43 MB 2025-02-15 09:12:17,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18922.60 MB 2025-02-15 09:12:17,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-15 09:12:17,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1855.98 MB 2025-02-15 09:12:17,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18896.60 MB 2025-02-15 09:12:17,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:12:17,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:12:17,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:12:17,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17319.28 MB 2025-02-15 09:12:17,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17620.33 MB 2025-02-15 09:12:17,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 301.05 MB 2025-02-15 09:12:17,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20778.58 MB 2025-02-15 09:12:17,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20940.06 MB 2025-02-15 09:12:17,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-15 09:12:17,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17906.64 MB 2025-02-15 09:12:17,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:12:17,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:12:17,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:12:17,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17782.40 MB 2025-02-15 09:12:17,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17998.62 MB 2025-02-15 09:12:17,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.23 MB 2025-02-15 09:12:17,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20940.06 MB 2025-02-15 09:12:17,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20940.06 MB 2025-02-15 09:12:17,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:17,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18018.17 MB 2025-02-15 09:12:17,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:12:17,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:12:17,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-15 09:12:17,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:17,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13515.71 MB 2025-02-15 09:12:17,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18199.38 MB 2025-02-15 09:12:17,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4683.67 MB 2025-02-15 09:12:17,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53150.22 MB 2025-02-15 09:12:17,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20940.06 MB 2025-02-15 09:12:17,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32210.16 MB 2025-02-15 09:12:17,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18199.38 MB 2025-02-15 09:12:18,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:12:18,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:12:18,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 09:12:18,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:18,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18199.38 MB 2025-02-15 09:12:18,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17368.36 MB 2025-02-15 09:12:18,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -831.02 MB 2025-02-15 09:12:18,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20940.06 MB 2025-02-15 09:12:18,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20940.06 MB 2025-02-15 09:12:18,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:18,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19001.83 MB 2025-02-15 09:12:18,238 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 09:12:18,238 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:12:18,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:12:18,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:12:18,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:12:18,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:18,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17368.36 MB 2025-02-15 09:12:18,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25794.54 MB 2025-02-15 09:12:18,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 09:12:18,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20940.06 MB 2025-02-15 09:12:18,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31411.14 MB 2025-02-15 09:12:18,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 09:12:18,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25794.54 MB 2025-02-15 09:12:18,401 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 09:12:18,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:18,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:12:18,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:18,403 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:12:18,408 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:12:18,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:18,409 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:12:18,409 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:12:28,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:28,265 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:12:28,273 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:12:28,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:28,279 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1316, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:12:28,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:28,281 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1316, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:12:48,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:12:48,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:12:48,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.33 seconds 2025-02-15 09:12:48,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:48,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22138.81 MB 2025-02-15 09:12:48,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26796.58 MB 2025-02-15 09:12:48,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4657.77 MB 2025-02-15 09:12:48,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39787.17 MB 2025-02-15 09:12:48,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37887.15 MB 2025-02-15 09:12:48,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1900.02 MB 2025-02-15 09:12:48,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35687.04 MB 2025-02-15 09:12:48,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:12:48,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:12:48,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:12:48,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:48,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26796.58 MB 2025-02-15 09:12:48,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22619.32 MB 2025-02-15 09:12:48,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4177.26 MB 2025-02-15 09:12:48,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37887.15 MB 2025-02-15 09:12:48,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 09:12:48,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2638.22 MB 2025-02-15 09:12:48,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36253.52 MB 2025-02-15 09:12:50,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:12:50,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:12:50,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:12:50,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:50,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22619.32 MB 2025-02-15 09:12:50,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23150.17 MB 2025-02-15 09:12:50,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:12:50,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-15 09:12:50,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-15 09:12:50,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11484.00 MB 2025-02-15 09:12:50,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27128.71 MB 2025-02-15 09:12:50,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:12:50,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:12:50,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:12:50,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:50,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23150.17 MB 2025-02-15 09:12:50,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25039.70 MB 2025-02-15 09:12:50,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:12:50,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 09:12:50,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-15 09:12:50,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:50,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26457.13 MB 2025-02-15 09:12:50,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:12:50,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:12:50,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:12:50,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:50,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25039.70 MB 2025-02-15 09:12:50,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27281.56 MB 2025-02-15 09:12:50,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:12:50,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 09:12:50,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 09:12:50,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:12:50,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32825.84 MB 2025-02-15 09:12:50,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:12:50,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:12:50,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:12:50,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:50,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23150.17 MB 2025-02-15 09:12:50,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27281.56 MB 2025-02-15 09:12:50,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:12:50,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-15 09:12:50,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34703.67 MB 2025-02-15 09:12:50,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:12:50,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32825.84 MB 2025-02-15 09:12:51,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:12:51,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:12:51,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:12:51,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:51,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28815.10 MB 2025-02-15 09:12:51,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29582.10 MB 2025-02-15 09:12:51,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:12:51,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34703.67 MB 2025-02-15 09:12:51,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 09:12:51,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:12:51,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.89 MB 2025-02-15 09:12:51,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:12:51,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:12:51,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:12:51,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:51,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29994.99 MB 2025-02-15 09:12:51,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30224.77 MB 2025-02-15 09:12:51,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.78 MB 2025-02-15 09:12:51,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 09:12:51,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 09:12:51,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:51,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30433.08 MB 2025-02-15 09:12:51,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:12:51,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:12:51,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.74 seconds 2025-02-15 09:12:51,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:51,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17553.76 MB 2025-02-15 09:12:51,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30425.85 MB 2025-02-15 09:12:51,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12872.09 MB 2025-02-15 09:12:51,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39787.17 MB 2025-02-15 09:12:51,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 09:12:51,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4666.16 MB 2025-02-15 09:12:51,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30433.08 MB 2025-02-15 09:12:51,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:12:51,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:12:51,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:12:51,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:51,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30425.85 MB 2025-02-15 09:12:51,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22558.15 MB 2025-02-15 09:12:51,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7867.70 MB 2025-02-15 09:12:51,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 09:12:51,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35121.00 MB 2025-02-15 09:12:51,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:12:51,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.51 MB 2025-02-15 09:12:51,309 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:12:51,309 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:12:51,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:12:51,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:12:51,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:12:51,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:12:51,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22558.15 MB 2025-02-15 09:12:51,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30997.17 MB 2025-02-15 09:12:51,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:12:51,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35121.00 MB 2025-02-15 09:12:51,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43511.71 MB 2025-02-15 09:12:51,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:12:51,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30997.17 MB 2025-02-15 09:12:51,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:12:51,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:51,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:12:51,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:51,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:12:51,482 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:12:51,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:12:51,483 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:12:51,483 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:13:18,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:18,260 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:13:18,265 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:13:18,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:18,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 151, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:13:18,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:18,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 151, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:13:20,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:13:20,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:13:20,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.36 seconds 2025-02-15 09:13:20,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:20,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14020.90 MB 2025-02-15 09:13:20,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14555.28 MB 2025-02-15 09:13:20,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 534.38 MB 2025-02-15 09:13:20,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56096.72 MB 2025-02-15 09:13:20,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 09:13:20,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38673.58 MB 2025-02-15 09:13:20,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23492.27 MB 2025-02-15 09:13:20,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:13:20,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:13:20,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:13:20,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:20,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14555.28 MB 2025-02-15 09:13:20,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14772.70 MB 2025-02-15 09:13:20,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.42 MB 2025-02-15 09:13:20,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 09:13:20,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17936.94 MB 2025-02-15 09:13:20,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 513.80 MB 2025-02-15 09:13:20,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16592.68 MB 2025-02-15 09:13:21,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:13:21,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:13:21,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-15 09:13:21,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14772.70 MB 2025-02-15 09:13:21,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14965.13 MB 2025-02-15 09:13:21,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 09:13:21,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17936.94 MB 2025-02-15 09:13:21,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17936.94 MB 2025-02-15 09:13:21,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:13:21,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18943.39 MB 2025-02-15 09:13:21,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:13:21,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:13:21,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:13:21,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14965.07 MB 2025-02-15 09:13:21,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15649.86 MB 2025-02-15 09:13:21,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 09:13:21,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17936.94 MB 2025-02-15 09:13:21,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17936.94 MB 2025-02-15 09:13:21,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:13:21,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16163.68 MB 2025-02-15 09:13:21,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:13:21,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:13:21,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:13:21,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15649.86 MB 2025-02-15 09:13:21,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16462.57 MB 2025-02-15 09:13:21,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 09:13:21,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17936.94 MB 2025-02-15 09:13:21,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19484.64 MB 2025-02-15 09:13:21,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1547.70 MB 2025-02-15 09:13:21,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18473.25 MB 2025-02-15 09:13:21,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:13:21,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:13:21,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:13:21,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14965.07 MB 2025-02-15 09:13:21,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16462.57 MB 2025-02-15 09:13:21,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 09:13:21,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17936.94 MB 2025-02-15 09:13:21,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19484.64 MB 2025-02-15 09:13:21,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1547.70 MB 2025-02-15 09:13:21,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18473.25 MB 2025-02-15 09:13:21,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:13:21,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:13:21,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:13:21,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17019.40 MB 2025-02-15 09:13:21,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17297.44 MB 2025-02-15 09:13:21,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 09:13:21,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19484.64 MB 2025-02-15 09:13:21,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19633.54 MB 2025-02-15 09:13:21,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-15 09:13:21,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17564.05 MB 2025-02-15 09:13:21,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:13:21,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:13:21,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:13:21,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17447.12 MB 2025-02-15 09:13:21,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17673.59 MB 2025-02-15 09:13:21,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.47 MB 2025-02-15 09:13:21,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19633.54 MB 2025-02-15 09:13:21,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19633.54 MB 2025-02-15 09:13:21,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:13:21,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17676.45 MB 2025-02-15 09:13:21,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:13:21,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:13:21,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-15 09:13:21,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13494.80 MB 2025-02-15 09:13:21,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17874.66 MB 2025-02-15 09:13:21,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4379.86 MB 2025-02-15 09:13:21,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56096.72 MB 2025-02-15 09:13:21,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19633.54 MB 2025-02-15 09:13:21,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36463.18 MB 2025-02-15 09:13:21,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.66 MB 2025-02-15 09:13:21,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:13:21,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:13:21,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:13:21,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17874.66 MB 2025-02-15 09:13:21,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17296.69 MB 2025-02-15 09:13:21,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -577.97 MB 2025-02-15 09:13:21,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19633.54 MB 2025-02-15 09:13:21,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-15 09:13:21,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-15 09:13:21,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18980.06 MB 2025-02-15 09:13:21,845 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:13:21,845 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 09:13:21,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:13:21,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:13:21,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:13:21,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:13:21,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17296.69 MB 2025-02-15 09:13:21,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25735.71 MB 2025-02-15 09:13:21,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:13:21,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-15 09:13:21,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30257.71 MB 2025-02-15 09:13:21,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 09:13:21,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25735.71 MB 2025-02-15 09:13:22,012 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:13:22,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:22,013 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:13:22,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:22,014 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:13:22,019 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:13:22,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:13:22,020 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:13:22,020 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 09:15:24,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:24,800 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:15:24,805 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:15:24,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:24,810 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 495, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:15:24,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:24,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 495, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:15:32,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:15:32,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:15:32,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.57 seconds 2025-02-15 09:15:32,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:32,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16417.95 MB 2025-02-15 09:15:32,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18169.72 MB 2025-02-15 09:15:32,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1751.78 MB 2025-02-15 09:15:32,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42842.72 MB 2025-02-15 09:15:32,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22772.97 MB 2025-02-15 09:15:32,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20069.74 MB 2025-02-15 09:15:32,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27021.78 MB 2025-02-15 09:15:32,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:15:32,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:15:32,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 09:15:32,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:32,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18169.72 MB 2025-02-15 09:15:32,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18352.25 MB 2025-02-15 09:15:32,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 182.53 MB 2025-02-15 09:15:32,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22772.97 MB 2025-02-15 09:15:32,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27409.78 MB 2025-02-15 09:15:32,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4636.80 MB 2025-02-15 09:15:32,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25753.75 MB 2025-02-15 09:15:34,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:15:34,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:15:34,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-15 09:15:34,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18352.25 MB 2025-02-15 09:15:34,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18883.09 MB 2025-02-15 09:15:34,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:15:34,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27409.78 MB 2025-02-15 09:15:34,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 09:15:34,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2512.39 MB 2025-02-15 09:15:34,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22861.64 MB 2025-02-15 09:15:34,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:15:34,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:15:34,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:15:34,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18883.09 MB 2025-02-15 09:15:34,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20772.63 MB 2025-02-15 09:15:34,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:15:34,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 09:15:34,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 09:15:34,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:15:34,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22190.05 MB 2025-02-15 09:15:34,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:15:34,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:15:34,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:15:34,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20772.63 MB 2025-02-15 09:15:34,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23014.48 MB 2025-02-15 09:15:34,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:15:34,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 09:15:34,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30559.70 MB 2025-02-15 09:15:34,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:15:34,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28558.76 MB 2025-02-15 09:15:34,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:15:34,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:15:34,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:15:34,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18883.09 MB 2025-02-15 09:15:34,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23014.48 MB 2025-02-15 09:15:34,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:15:34,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 09:15:34,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30559.70 MB 2025-02-15 09:15:34,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:15:34,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28558.76 MB 2025-02-15 09:15:34,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:15:34,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:15:34,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:15:34,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24548.02 MB 2025-02-15 09:15:34,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25315.03 MB 2025-02-15 09:15:34,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:15:34,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30559.70 MB 2025-02-15 09:15:34,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30974.94 MB 2025-02-15 09:15:34,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:15:34,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26022.81 MB 2025-02-15 09:15:34,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:15:34,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:15:34,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:15:34,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25727.91 MB 2025-02-15 09:15:34,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25955.78 MB 2025-02-15 09:15:34,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.87 MB 2025-02-15 09:15:34,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30974.94 MB 2025-02-15 09:15:34,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30974.94 MB 2025-02-15 09:15:34,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:15:34,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26138.50 MB 2025-02-15 09:15:34,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:15:34,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:15:34,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.93 seconds 2025-02-15 09:15:34,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:34,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14693.33 MB 2025-02-15 09:15:34,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26156.85 MB 2025-02-15 09:15:34,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11463.53 MB 2025-02-15 09:15:34,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42842.72 MB 2025-02-15 09:15:34,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30974.94 MB 2025-02-15 09:15:34,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11867.78 MB 2025-02-15 09:15:34,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26156.85 MB 2025-02-15 09:15:35,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:15:35,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:15:35,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:15:35,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:35,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26156.85 MB 2025-02-15 09:15:35,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19697.72 MB 2025-02-15 09:15:35,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6459.14 MB 2025-02-15 09:15:35,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30974.94 MB 2025-02-15 09:15:35,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30974.94 MB 2025-02-15 09:15:35,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:15:35,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28668.52 MB 2025-02-15 09:15:35,032 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:15:35,033 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:15:35,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:15:35,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:15:35,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:15:35,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:15:35,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19697.72 MB 2025-02-15 09:15:35,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28136.74 MB 2025-02-15 09:15:35,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:15:35,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30974.94 MB 2025-02-15 09:15:35,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39365.64 MB 2025-02-15 09:15:35,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:15:35,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28136.74 MB 2025-02-15 09:15:35,206 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:15:35,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:35,207 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:15:35,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:35,208 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:15:35,213 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:15:35,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:15:35,214 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:15:35,215 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:17:29,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:17:29,034 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:17:29,040 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:17:29,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:17:29,044 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2934, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:17:29,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:17:29,045 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2934, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:18:14,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:18:14,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:18:14,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.23 seconds 2025-02-15 09:18:14,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:14,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33415.94 MB 2025-02-15 09:18:14,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43799.20 MB 2025-02-15 09:18:14,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10383.26 MB 2025-02-15 09:18:14,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72397.88 MB 2025-02-15 09:18:14,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47311.75 MB 2025-02-15 09:18:14,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25086.13 MB 2025-02-15 09:18:14,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54182.46 MB 2025-02-15 09:18:14,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:18:14,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:18:14,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 09:18:14,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:14,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43799.20 MB 2025-02-15 09:18:14,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31033.58 MB 2025-02-15 09:18:14,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12765.62 MB 2025-02-15 09:18:14,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47311.75 MB 2025-02-15 09:18:14,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69193.43 MB 2025-02-15 09:18:14,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21881.68 MB 2025-02-15 09:18:14,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73514.77 MB 2025-02-15 09:18:16,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:18:16,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:18:16,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:18:16,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31033.58 MB 2025-02-15 09:18:16,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31564.42 MB 2025-02-15 09:18:16,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:18:16,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69193.43 MB 2025-02-15 09:18:16,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33858.52 MB 2025-02-15 09:18:16,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35334.91 MB 2025-02-15 09:18:16,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35542.97 MB 2025-02-15 09:18:16,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:18:16,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:18:16,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:18:16,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31564.42 MB 2025-02-15 09:18:16,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33453.96 MB 2025-02-15 09:18:16,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:18:16,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33858.52 MB 2025-02-15 09:18:16,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-15 09:18:16,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:18:16,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34871.39 MB 2025-02-15 09:18:16,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:18:16,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:18:16,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:18:16,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33453.96 MB 2025-02-15 09:18:16,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35695.81 MB 2025-02-15 09:18:16,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:18:16,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36689.67 MB 2025-02-15 09:18:16,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42823.84 MB 2025-02-15 09:18:16,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:18:16,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41240.10 MB 2025-02-15 09:18:16,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:18:16,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:18:16,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:18:16,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31564.42 MB 2025-02-15 09:18:16,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35695.81 MB 2025-02-15 09:18:16,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:18:16,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33858.52 MB 2025-02-15 09:18:16,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42823.84 MB 2025-02-15 09:18:16,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:18:16,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41240.10 MB 2025-02-15 09:18:16,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:18:16,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:18:16,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:18:16,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37229.36 MB 2025-02-15 09:18:16,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37996.36 MB 2025-02-15 09:18:16,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:18:16,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42823.84 MB 2025-02-15 09:18:16,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43239.08 MB 2025-02-15 09:18:16,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:18:16,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38704.15 MB 2025-02-15 09:18:16,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:18:16,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:18:16,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:18:16,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38409.25 MB 2025-02-15 09:18:16,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38637.37 MB 2025-02-15 09:18:16,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-15 09:18:16,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43239.08 MB 2025-02-15 09:18:16,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43239.08 MB 2025-02-15 09:18:16,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:18:16,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38861.25 MB 2025-02-15 09:18:16,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:18:16,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:18:16,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.89 seconds 2025-02-15 09:18:16,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:16,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.32 MB 2025-02-15 09:18:16,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38837.51 MB 2025-02-15 09:18:16,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15645.19 MB 2025-02-15 09:18:16,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62174.27 MB 2025-02-15 09:18:16,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43239.08 MB 2025-02-15 09:18:16,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18935.19 MB 2025-02-15 09:18:16,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38861.25 MB 2025-02-15 09:18:17,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:18:17,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:18:17,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:18:17,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:17,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38837.51 MB 2025-02-15 09:18:17,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28182.95 MB 2025-02-15 09:18:17,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10654.56 MB 2025-02-15 09:18:17,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43239.08 MB 2025-02-15 09:18:17,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43239.08 MB 2025-02-15 09:18:17,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:18:17,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41337.50 MB 2025-02-15 09:18:17,229 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 09:18:17,229 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:18:17,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:18:17,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:18:17,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:18:17,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:18:17,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28182.95 MB 2025-02-15 09:18:17,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36582.86 MB 2025-02-15 09:18:17,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-15 09:18:17,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43239.08 MB 2025-02-15 09:18:17,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47414.51 MB 2025-02-15 09:18:17,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 09:18:17,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36582.86 MB 2025-02-15 09:18:17,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 09:18:17,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:18:17,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:18:17,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:18:17,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:18:17,403 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:18:17,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:18:17,404 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:18:17,404 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:19:57,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:19:57,194 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:19:57,199 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:19:57,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:19:57,203 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2369, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:19:57,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:19:57,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2369, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:20:33,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:20:33,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:20:33,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.63 seconds 2025-02-15 09:20:33,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:33,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29476.28 MB 2025-02-15 09:20:33,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37860.69 MB 2025-02-15 09:20:33,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8384.41 MB 2025-02-15 09:20:33,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55765.37 MB 2025-02-15 09:20:33,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43274.73 MB 2025-02-15 09:20:33,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12490.64 MB 2025-02-15 09:20:33,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46874.88 MB 2025-02-15 09:20:34,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:20:34,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:20:34,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 09:20:34,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:34,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37860.69 MB 2025-02-15 09:20:34,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28093.54 MB 2025-02-15 09:20:34,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9767.15 MB 2025-02-15 09:20:34,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43274.73 MB 2025-02-15 09:20:34,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59020.15 MB 2025-02-15 09:20:34,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15745.42 MB 2025-02-15 09:20:34,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59414.37 MB 2025-02-15 09:20:36,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:20:36,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:20:36,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 09:20:36,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28093.54 MB 2025-02-15 09:20:36,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28624.38 MB 2025-02-15 09:20:36,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:20:36,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59020.15 MB 2025-02-15 09:20:36,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:20:36,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22714.25 MB 2025-02-15 09:20:36,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32602.93 MB 2025-02-15 09:20:36,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:20:36,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:20:36,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:20:36,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.38 MB 2025-02-15 09:20:36,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30513.92 MB 2025-02-15 09:20:36,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:20:36,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:20:36,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:20:36,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:20:36,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31931.35 MB 2025-02-15 09:20:36,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:20:36,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:20:36,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:20:36,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30513.92 MB 2025-02-15 09:20:36,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32755.77 MB 2025-02-15 09:20:36,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:20:36,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:20:36,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40552.63 MB 2025-02-15 09:20:36,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 09:20:36,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.06 MB 2025-02-15 09:20:36,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:20:36,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:20:36,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:20:36,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.38 MB 2025-02-15 09:20:36,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32755.77 MB 2025-02-15 09:20:36,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:20:36,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:20:36,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40552.63 MB 2025-02-15 09:20:36,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 09:20:36,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.06 MB 2025-02-15 09:20:36,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:20:36,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:20:36,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:20:36,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34289.32 MB 2025-02-15 09:20:36,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35056.32 MB 2025-02-15 09:20:36,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:20:36,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40552.63 MB 2025-02-15 09:20:36,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40969.96 MB 2025-02-15 09:20:36,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:20:36,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35764.11 MB 2025-02-15 09:20:36,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:20:36,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:20:36,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:20:36,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35469.21 MB 2025-02-15 09:20:36,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.25 MB 2025-02-15 09:20:36,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.04 MB 2025-02-15 09:20:36,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40969.96 MB 2025-02-15 09:20:36,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40969.96 MB 2025-02-15 09:20:36,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:20:36,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.44 MB 2025-02-15 09:20:36,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:20:36,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:20:36,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.23 seconds 2025-02-15 09:20:36,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21222.49 MB 2025-02-15 09:20:36,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35897.34 MB 2025-02-15 09:20:36,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14674.84 MB 2025-02-15 09:20:36,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55765.37 MB 2025-02-15 09:20:36,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40969.96 MB 2025-02-15 09:20:36,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14795.41 MB 2025-02-15 09:20:36,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.44 MB 2025-02-15 09:20:36,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:20:36,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:20:36,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:20:36,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35897.34 MB 2025-02-15 09:20:36,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26212.41 MB 2025-02-15 09:20:36,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9684.93 MB 2025-02-15 09:20:36,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40969.96 MB 2025-02-15 09:20:36,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40969.96 MB 2025-02-15 09:20:36,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:20:36,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38397.48 MB 2025-02-15 09:20:36,727 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 09:20:36,727 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:20:36,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:20:36,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:20:36,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:20:36,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:20:36,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26212.41 MB 2025-02-15 09:20:36,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34609.81 MB 2025-02-15 09:20:36,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 09:20:36,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40969.96 MB 2025-02-15 09:20:36,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40969.96 MB 2025-02-15 09:20:36,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:20:36,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34609.81 MB 2025-02-15 09:20:36,896 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 09:20:36,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:36,897 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:20:36,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:36,898 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:20:36,903 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:20:36,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:36,904 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:20:36,904 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:20:44,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:44,522 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:20:44,527 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:20:44,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:44,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2222, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:20:44,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:20:44,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2222, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:21:19,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:21:19,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:21:19,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.78 seconds 2025-02-15 09:21:19,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:19,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28451.96 MB 2025-02-15 09:21:19,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36316.28 MB 2025-02-15 09:21:19,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7864.32 MB 2025-02-15 09:21:19,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49320.82 MB 2025-02-15 09:21:19,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41043.36 MB 2025-02-15 09:21:19,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8277.46 MB 2025-02-15 09:21:19,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45171.09 MB 2025-02-15 09:21:19,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:21:19,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:21:19,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 09:21:19,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:19,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36316.28 MB 2025-02-15 09:21:19,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27330.38 MB 2025-02-15 09:21:19,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8985.89 MB 2025-02-15 09:21:19,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41043.36 MB 2025-02-15 09:21:19,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58376.32 MB 2025-02-15 09:21:19,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17332.96 MB 2025-02-15 09:21:19,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59187.25 MB 2025-02-15 09:21:21,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:21:21,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:21:21,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:21:21,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27330.38 MB 2025-02-15 09:21:21,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27861.23 MB 2025-02-15 09:21:21,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:21:21,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58376.32 MB 2025-02-15 09:21:21,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31128.03 MB 2025-02-15 09:21:21,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27248.30 MB 2025-02-15 09:21:21,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.77 MB 2025-02-15 09:21:21,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:21:21,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:21:21,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:21:21,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27861.23 MB 2025-02-15 09:21:21,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29750.76 MB 2025-02-15 09:21:21,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:21:21,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31128.03 MB 2025-02-15 09:21:21,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33959.18 MB 2025-02-15 09:21:21,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:21:21,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31168.19 MB 2025-02-15 09:21:21,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:21:21,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:21:21,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:21:21,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29750.76 MB 2025-02-15 09:21:21,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31992.62 MB 2025-02-15 09:21:21,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:21:21,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33959.18 MB 2025-02-15 09:21:21,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39621.49 MB 2025-02-15 09:21:21,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:21:21,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37536.90 MB 2025-02-15 09:21:21,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:21:21,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:21:21,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:21:21,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27861.23 MB 2025-02-15 09:21:21,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31992.62 MB 2025-02-15 09:21:21,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:21:21,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31128.03 MB 2025-02-15 09:21:21,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39621.49 MB 2025-02-15 09:21:21,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 09:21:21,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37536.90 MB 2025-02-15 09:21:21,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:21:21,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:21:21,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:21:21,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33526.16 MB 2025-02-15 09:21:21,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34293.16 MB 2025-02-15 09:21:21,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:21:21,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39621.49 MB 2025-02-15 09:21:21,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 09:21:21,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:21:21,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35000.95 MB 2025-02-15 09:21:21,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:21:21,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:21:21,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:21:21,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34706.05 MB 2025-02-15 09:21:21,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34933.56 MB 2025-02-15 09:21:21,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.51 MB 2025-02-15 09:21:21,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 09:21:21,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 09:21:21,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:21,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35147.13 MB 2025-02-15 09:21:21,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:21:21,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:21:21,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.39 seconds 2025-02-15 09:21:21,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:21,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20710.33 MB 2025-02-15 09:21:21,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35134.12 MB 2025-02-15 09:21:21,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14423.78 MB 2025-02-15 09:21:21,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49320.82 MB 2025-02-15 09:21:21,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 09:21:21,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9281.99 MB 2025-02-15 09:21:21,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35147.13 MB 2025-02-15 09:21:22,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:21:22,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:21:22,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:21:22,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:22,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35134.12 MB 2025-02-15 09:21:22,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25707.02 MB 2025-02-15 09:21:22,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9427.10 MB 2025-02-15 09:21:22,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 09:21:22,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-15 09:21:22,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:22,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37639.63 MB 2025-02-15 09:21:22,209 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 09:21:22,209 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:21:22,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:21:22,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:21:22,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:21:22,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:22,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25707.02 MB 2025-02-15 09:21:22,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34124.76 MB 2025-02-15 09:21:22,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-15 09:21:22,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40038.83 MB 2025-02-15 09:21:22,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48406.46 MB 2025-02-15 09:21:22,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 09:21:22,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34124.76 MB 2025-02-15 09:21:22,374 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 09:21:22,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:22,375 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:21:22,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:22,376 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:21:22,381 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:21:22,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:22,382 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:21:22,382 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:21:31,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:31,102 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:21:31,107 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:21:31,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:31,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 140, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:21:31,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:31,111 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 140, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:21:33,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:21:33,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:21:33,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.22 seconds 2025-02-15 09:21:33,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:33,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.25 MB 2025-02-15 09:21:33,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14439.70 MB 2025-02-15 09:21:33,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 495.45 MB 2025-02-15 09:21:33,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56774.10 MB 2025-02-15 09:21:33,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22307.41 MB 2025-02-15 09:21:33,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34466.69 MB 2025-02-15 09:21:33,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23415.62 MB 2025-02-15 09:21:33,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:21:33,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:21:33,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:21:33,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:33,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14439.70 MB 2025-02-15 09:21:33,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14679.75 MB 2025-02-15 09:21:33,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.05 MB 2025-02-15 09:21:33,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22307.41 MB 2025-02-15 09:21:33,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22307.41 MB 2025-02-15 09:21:33,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:33,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16406.21 MB 2025-02-15 09:21:34,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:21:34,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:21:34,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.68 seconds 2025-02-15 09:21:34,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14679.75 MB 2025-02-15 09:21:34,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14865.54 MB 2025-02-15 09:21:34,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.79 MB 2025-02-15 09:21:34,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22307.41 MB 2025-02-15 09:21:34,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21835.55 MB 2025-02-15 09:21:34,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 09:21:34,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18849.40 MB 2025-02-15 09:21:34,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:21:34,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:21:34,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:21:34,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-15 09:21:34,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15526.65 MB 2025-02-15 09:21:34,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.18 MB 2025-02-15 09:21:34,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21835.55 MB 2025-02-15 09:21:34,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21835.55 MB 2025-02-15 09:21:34,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:34,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16022.76 MB 2025-02-15 09:21:34,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:21:34,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:21:34,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 09:21:34,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.65 MB 2025-02-15 09:21:34,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-15 09:21:34,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.69 MB 2025-02-15 09:21:34,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21835.55 MB 2025-02-15 09:21:34,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21835.55 MB 2025-02-15 09:21:34,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:34,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18251.80 MB 2025-02-15 09:21:34,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:21:34,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:21:34,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:21:34,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-15 09:21:34,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-15 09:21:34,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.87 MB 2025-02-15 09:21:34,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21835.55 MB 2025-02-15 09:21:34,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21835.55 MB 2025-02-15 09:21:34,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:34,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18251.80 MB 2025-02-15 09:21:34,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:21:34,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:21:34,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 09:21:34,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16848.08 MB 2025-02-15 09:21:34,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17116.53 MB 2025-02-15 09:21:34,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.45 MB 2025-02-15 09:21:34,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21835.55 MB 2025-02-15 09:21:34,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21978.15 MB 2025-02-15 09:21:34,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 142.61 MB 2025-02-15 09:21:34,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17374.57 MB 2025-02-15 09:21:34,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:21:34,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:21:34,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:21:34,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17261.05 MB 2025-02-15 09:21:34,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17472.23 MB 2025-02-15 09:21:34,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.18 MB 2025-02-15 09:21:34,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21978.15 MB 2025-02-15 09:21:34,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21978.15 MB 2025-02-15 09:21:34,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:34,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17476.72 MB 2025-02-15 09:21:34,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:21:34,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:21:34,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-15 09:21:34,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13456.48 MB 2025-02-15 09:21:34,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17673.06 MB 2025-02-15 09:21:34,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4216.58 MB 2025-02-15 09:21:34,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56774.10 MB 2025-02-15 09:21:34,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21978.15 MB 2025-02-15 09:21:34,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34795.95 MB 2025-02-15 09:21:34,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17673.06 MB 2025-02-15 09:21:34,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:21:34,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:21:34,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:21:34,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17673.06 MB 2025-02-15 09:21:34,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20683.41 MB 2025-02-15 09:21:34,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.35 MB 2025-02-15 09:21:34,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21978.15 MB 2025-02-15 09:21:34,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21978.15 MB 2025-02-15 09:21:34,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:21:34,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20984.41 MB 2025-02-15 09:21:34,459 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 09:21:34,460 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 09:21:34,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:21:34,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:21:34,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:21:34,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:21:34,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20683.41 MB 2025-02-15 09:21:34,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29112.53 MB 2025-02-15 09:21:34,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 09:21:34,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21978.15 MB 2025-02-15 09:21:34,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32453.43 MB 2025-02-15 09:21:34,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 09:21:34,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29112.53 MB 2025-02-15 09:21:34,623 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 09:21:34,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:34,625 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:21:34,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:34,626 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:21:34,631 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:21:34,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:21:34,632 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:21:34,632 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 09:23:31,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:31,857 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:23:31,862 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:23:31,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:31,866 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:23:31,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:31,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:23:34,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:23:34,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:23:34,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.35 seconds 2025-02-15 09:23:34,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:34,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-15 09:23:34,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-15 09:23:34,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 09:23:34,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40835.74 MB 2025-02-15 09:23:34,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:34,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17618.17 MB 2025-02-15 09:23:34,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-15 09:23:34,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:23:34,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:23:34,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:23:34,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:34,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-15 09:23:34,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.49 MB 2025-02-15 09:23:34,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-15 09:23:34,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:34,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:34,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:34,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16641.13 MB 2025-02-15 09:23:34,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:23:34,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:23:34,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 09:23:34,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:34,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.49 MB 2025-02-15 09:23:34,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14991.58 MB 2025-02-15 09:23:34,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-15 09:23:34,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:34,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:34,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:34,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18966.14 MB 2025-02-15 09:23:34,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:23:34,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:23:34,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:23:34,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:34,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-15 09:23:34,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15685.75 MB 2025-02-15 09:23:34,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-15 09:23:34,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:34,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:34,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:34,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16206.66 MB 2025-02-15 09:23:35,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:23:35,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:23:35,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:23:35,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15685.75 MB 2025-02-15 09:23:35,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-15 09:23:35,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-15 09:23:35,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:35,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:35,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:35,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-15 09:23:35,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:23:35,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:23:35,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:23:35,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-15 09:23:35,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-15 09:23:35,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-15 09:23:35,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:35,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23217.57 MB 2025-02-15 09:23:35,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:35,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-15 09:23:35,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:23:35,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:23:35,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:23:35,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17073.25 MB 2025-02-15 09:23:35,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17355.12 MB 2025-02-15 09:23:35,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-15 09:23:35,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23217.57 MB 2025-02-15 09:23:35,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23370.66 MB 2025-02-15 09:23:35,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 09:23:35,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17625.30 MB 2025-02-15 09:23:35,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:23:35,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:23:35,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:23:35,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17506.86 MB 2025-02-15 09:23:35,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17715.44 MB 2025-02-15 09:23:35,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.58 MB 2025-02-15 09:23:35,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23370.66 MB 2025-02-15 09:23:35,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23372.76 MB 2025-02-15 09:23:35,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 09:23:35,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17721.05 MB 2025-02-15 09:23:35,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:23:35,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:23:35,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-15 09:23:35,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-15 09:23:35,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17916.49 MB 2025-02-15 09:23:35,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4414.72 MB 2025-02-15 09:23:35,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40835.74 MB 2025-02-15 09:23:35,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23372.76 MB 2025-02-15 09:23:35,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17462.98 MB 2025-02-15 09:23:35,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17916.49 MB 2025-02-15 09:23:35,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:23:35,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:23:35,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:23:35,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17916.49 MB 2025-02-15 09:23:35,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17311.80 MB 2025-02-15 09:23:35,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -604.69 MB 2025-02-15 09:23:35,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23372.76 MB 2025-02-15 09:23:35,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23372.76 MB 2025-02-15 09:23:35,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:23:35,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19021.49 MB 2025-02-15 09:23:35,402 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 09:23:35,402 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 09:23:35,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:23:35,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:23:35,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:23:35,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:23:35,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17311.80 MB 2025-02-15 09:23:35,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25750.63 MB 2025-02-15 09:23:35,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 09:23:35,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23372.76 MB 2025-02-15 09:23:35,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31761.37 MB 2025-02-15 09:23:35,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 09:23:35,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25750.63 MB 2025-02-15 09:23:35,568 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 09:23:35,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:35,570 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:23:35,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:35,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:23:35,575 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:23:35,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:35,577 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:23:35,577 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 09:23:42,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:42,558 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:23:42,563 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:23:42,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:42,567 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2492, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:23:42,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:23:42,567 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2492, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:24:21,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:24:21,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:24:21,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.56 seconds 2025-02-15 09:24:21,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:21,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30334.93 MB 2025-02-15 09:24:21,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39153.98 MB 2025-02-15 09:24:21,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8819.05 MB 2025-02-15 09:24:21,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57518.59 MB 2025-02-15 09:24:21,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42666.56 MB 2025-02-15 09:24:21,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14852.03 MB 2025-02-15 09:24:21,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47973.03 MB 2025-02-15 09:24:21,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:24:21,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:24:21,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:24:21,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:21,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39153.98 MB 2025-02-15 09:24:21,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28734.82 MB 2025-02-15 09:24:21,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10419.17 MB 2025-02-15 09:24:21,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42666.56 MB 2025-02-15 09:24:21,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61306.04 MB 2025-02-15 09:24:21,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18639.49 MB 2025-02-15 09:24:21,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63990.27 MB 2025-02-15 09:24:23,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:24:23,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:24:23,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:24:23,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28734.82 MB 2025-02-15 09:24:23,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29265.66 MB 2025-02-15 09:24:23,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:24:23,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61306.04 MB 2025-02-15 09:24:23,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31560.04 MB 2025-02-15 09:24:23,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29746.00 MB 2025-02-15 09:24:23,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33244.21 MB 2025-02-15 09:24:23,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:24:23,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:24:23,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:24:23,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29265.66 MB 2025-02-15 09:24:23,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31155.19 MB 2025-02-15 09:24:23,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:24:23,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31560.04 MB 2025-02-15 09:24:23,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34391.20 MB 2025-02-15 09:24:23,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:24:23,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32572.62 MB 2025-02-15 09:24:23,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:24:23,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:24:23,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:24:23,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31155.19 MB 2025-02-15 09:24:23,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33397.05 MB 2025-02-15 09:24:23,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:24:23,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34391.20 MB 2025-02-15 09:24:23,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 09:24:23,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:24:23,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38941.33 MB 2025-02-15 09:24:23,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:24:23,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:24:23,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:24:23,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29265.66 MB 2025-02-15 09:24:23,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33397.05 MB 2025-02-15 09:24:23,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:24:23,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31560.04 MB 2025-02-15 09:24:23,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-15 09:24:23,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:24:23,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38941.33 MB 2025-02-15 09:24:23,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:24:23,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:24:23,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:24:23,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34930.59 MB 2025-02-15 09:24:23,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.59 MB 2025-02-15 09:24:23,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:24:23,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-15 09:24:23,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-15 09:24:23,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:24:23,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36405.38 MB 2025-02-15 09:24:23,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:24:23,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:24:23,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:24:23,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36110.48 MB 2025-02-15 09:24:23,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36340.35 MB 2025-02-15 09:24:23,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.87 MB 2025-02-15 09:24:23,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40942.70 MB 2025-02-15 09:24:23,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-15 09:24:23,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:24:23,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36554.07 MB 2025-02-15 09:24:23,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:24:23,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:24:23,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.15 seconds 2025-02-15 09:24:23,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21651.82 MB 2025-02-15 09:24:23,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36541.23 MB 2025-02-15 09:24:23,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14889.41 MB 2025-02-15 09:24:23,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48834.28 MB 2025-02-15 09:24:23,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-15 09:24:23,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7891.58 MB 2025-02-15 09:24:23,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36554.07 MB 2025-02-15 09:24:23,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:24:23,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:24:23,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:24:23,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:23,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36541.23 MB 2025-02-15 09:24:23,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26653.16 MB 2025-02-15 09:24:23,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9888.07 MB 2025-02-15 09:24:23,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40942.70 MB 2025-02-15 09:24:23,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-15 09:24:23,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:24:23,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39050.44 MB 2025-02-15 09:24:24,011 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 09:24:24,011 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:24:24,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:24:24,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:24:24,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:24:24,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:24:24,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26653.16 MB 2025-02-15 09:24:24,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35083.59 MB 2025-02-15 09:24:24,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.43 MB 2025-02-15 09:24:24,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40942.70 MB 2025-02-15 09:24:24,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45134.91 MB 2025-02-15 09:24:24,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 09:24:24,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35083.59 MB 2025-02-15 09:24:24,179 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 09:24:24,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:24:24,180 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:24:24,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:24:24,181 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:24:24,186 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:24:24,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:24:24,187 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:24:24,187 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:25:32,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:32,101 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:25:32,106 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:25:32,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:32,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 90, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:25:32,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:32,111 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 90, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:25:33,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:25:33,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:25:33,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.41 seconds 2025-02-15 09:25:33,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:33,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13595.84 MB 2025-02-15 09:25:33,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13914.35 MB 2025-02-15 09:25:33,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.50 MB 2025-02-15 09:25:33,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53515.12 MB 2025-02-15 09:25:33,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:33,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36563.85 MB 2025-02-15 09:25:33,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22840.72 MB 2025-02-15 09:25:33,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:25:33,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:25:33,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:25:33,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:33,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13914.35 MB 2025-02-15 09:25:33,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14068.66 MB 2025-02-15 09:25:33,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 154.31 MB 2025-02-15 09:25:33,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:33,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:33,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:33,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14546.48 MB 2025-02-15 09:25:33,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:25:33,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:25:33,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.44 seconds 2025-02-15 09:25:33,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:33,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14068.66 MB 2025-02-15 09:25:33,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14188.10 MB 2025-02-15 09:25:33,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 119.44 MB 2025-02-15 09:25:33,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:33,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:33,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:33,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18154.41 MB 2025-02-15 09:25:33,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:25:33,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:25:33,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:25:33,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:33,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.03 MB 2025-02-15 09:25:33,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14613.08 MB 2025-02-15 09:25:33,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.04 MB 2025-02-15 09:25:33,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:33,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:33,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:33,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14932.00 MB 2025-02-15 09:25:34,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:25:34,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:25:34,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:25:34,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14613.08 MB 2025-02-15 09:25:34,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15129.34 MB 2025-02-15 09:25:34,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.26 MB 2025-02-15 09:25:34,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:34,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:34,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:34,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16364.96 MB 2025-02-15 09:25:34,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:25:34,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:25:34,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:25:34,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.03 MB 2025-02-15 09:25:34,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15129.34 MB 2025-02-15 09:25:34,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 941.31 MB 2025-02-15 09:25:34,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:34,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16951.28 MB 2025-02-15 09:25:34,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:34,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16364.96 MB 2025-02-15 09:25:34,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:25:34,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:25:34,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:25:34,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15627.74 MB 2025-02-15 09:25:34,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15845.07 MB 2025-02-15 09:25:34,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.33 MB 2025-02-15 09:25:34,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16951.28 MB 2025-02-15 09:25:34,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17087.59 MB 2025-02-15 09:25:34,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-15 09:25:34,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16004.32 MB 2025-02-15 09:25:34,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:25:34,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:25:34,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:25:34,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15982.21 MB 2025-02-15 09:25:34,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16197.31 MB 2025-02-15 09:25:34,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 215.10 MB 2025-02-15 09:25:34,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17087.59 MB 2025-02-15 09:25:34,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17087.59 MB 2025-02-15 09:25:34,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:25:34,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16197.31 MB 2025-02-15 09:25:34,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:25:34,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:25:34,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.01 seconds 2025-02-15 09:25:34,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13282.27 MB 2025-02-15 09:25:34,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16389.07 MB 2025-02-15 09:25:34,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3106.79 MB 2025-02-15 09:25:34,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53515.12 MB 2025-02-15 09:25:34,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17087.59 MB 2025-02-15 09:25:34,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36427.53 MB 2025-02-15 09:25:34,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16389.07 MB 2025-02-15 09:25:34,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:25:34,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:25:34,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 09:25:34,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13805.49 MB 2025-02-15 09:25:34,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16679.80 MB 2025-02-15 09:25:34,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2874.32 MB 2025-02-15 09:25:34,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17087.59 MB 2025-02-15 09:25:34,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17983.08 MB 2025-02-15 09:25:34,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 895.48 MB 2025-02-15 09:25:34,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16967.38 MB 2025-02-15 09:25:34,397 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7783, cut from 7785 2025-02-15 09:25:34,397 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 09:25:34,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:25:34,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:25:34,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:25:34,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:25:34,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16679.80 MB 2025-02-15 09:25:34,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24727.29 MB 2025-02-15 09:25:34,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8047.49 MB 2025-02-15 09:25:34,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17983.08 MB 2025-02-15 09:25:34,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27986.49 MB 2025-02-15 09:25:34,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10003.42 MB 2025-02-15 09:25:34,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24727.29 MB 2025-02-15 09:25:34,555 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7575] 2025-02-15 09:25:34,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:34,556 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:25:34,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:34,557 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:25:34,562 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:25:34,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:34,563 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:25:34,563 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 09:25:49,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:49,376 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:25:49,383 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:25:49,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:49,390 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1533, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:25:49,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:25:49,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1533, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:26:13,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:26:13,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:26:13,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.72 seconds 2025-02-15 09:26:13,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:13,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23650.90 MB 2025-02-15 09:26:13,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29076.23 MB 2025-02-15 09:26:13,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5425.33 MB 2025-02-15 09:26:13,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35989.23 MB 2025-02-15 09:26:13,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-15 09:26:13,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1920.99 MB 2025-02-15 09:26:13,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37878.61 MB 2025-02-15 09:26:13,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:26:13,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:26:13,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:26:13,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:13,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29076.23 MB 2025-02-15 09:26:13,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23747.44 MB 2025-02-15 09:26:13,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5328.79 MB 2025-02-15 09:26:13,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-15 09:26:13,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49306.14 MB 2025-02-15 09:26:13,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11395.92 MB 2025-02-15 09:26:13,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44087.34 MB 2025-02-15 09:26:15,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:26:15,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:26:15,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:26:15,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23747.44 MB 2025-02-15 09:26:15,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24278.28 MB 2025-02-15 09:26:15,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:26:15,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49306.14 MB 2025-02-15 09:26:15,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33900.46 MB 2025-02-15 09:26:15,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15405.68 MB 2025-02-15 09:26:15,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28256.83 MB 2025-02-15 09:26:15,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:26:15,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:26:15,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:26:15,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-15 09:26:15,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26167.82 MB 2025-02-15 09:26:15,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:26:15,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33900.46 MB 2025-02-15 09:26:15,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33900.46 MB 2025-02-15 09:26:15,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:26:15,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27585.24 MB 2025-02-15 09:26:15,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:26:15,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:26:15,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:26:15,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26167.82 MB 2025-02-15 09:26:15,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-15 09:26:15,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:26:15,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33900.46 MB 2025-02-15 09:26:15,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36731.62 MB 2025-02-15 09:26:15,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:26:15,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-15 09:26:15,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:26:15,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:26:15,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:26:15,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-15 09:26:15,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-15 09:26:15,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:26:15,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33900.46 MB 2025-02-15 09:26:15,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36731.62 MB 2025-02-15 09:26:15,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:26:15,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-15 09:26:15,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:26:15,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:26:15,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:26:15,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29943.21 MB 2025-02-15 09:26:15,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30710.22 MB 2025-02-15 09:26:15,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:26:15,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36731.62 MB 2025-02-15 09:26:15,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37146.85 MB 2025-02-15 09:26:15,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:26:15,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31418.00 MB 2025-02-15 09:26:15,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:26:15,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:26:15,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:26:15,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31123.10 MB 2025-02-15 09:26:15,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31351.50 MB 2025-02-15 09:26:15,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-15 09:26:15,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37146.85 MB 2025-02-15 09:26:15,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37146.85 MB 2025-02-15 09:26:15,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:26:15,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31540.19 MB 2025-02-15 09:26:15,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:26:15,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:26:15,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.15 seconds 2025-02-15 09:26:15,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18309.80 MB 2025-02-15 09:26:15,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31552.35 MB 2025-02-15 09:26:15,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13242.55 MB 2025-02-15 09:26:15,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35989.23 MB 2025-02-15 09:26:15,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37146.85 MB 2025-02-15 09:26:15,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1157.63 MB 2025-02-15 09:26:15,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31552.35 MB 2025-02-15 09:26:15,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:26:15,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:26:15,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:26:15,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31552.35 MB 2025-02-15 09:26:15,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23302.92 MB 2025-02-15 09:26:15,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8249.43 MB 2025-02-15 09:26:15,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37146.85 MB 2025-02-15 09:26:15,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37146.85 MB 2025-02-15 09:26:15,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:26:15,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34054.50 MB 2025-02-15 09:26:15,833 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 09:26:15,833 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:26:15,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:26:15,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:26:15,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:26:15,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:15,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23302.92 MB 2025-02-15 09:26:15,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31710.66 MB 2025-02-15 09:26:15,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 09:26:15,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37146.85 MB 2025-02-15 09:26:15,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45506.10 MB 2025-02-15 09:26:15,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 09:26:15,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31710.66 MB 2025-02-15 09:26:16,001 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 09:26:16,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:16,002 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:26:16,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:16,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:26:16,007 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:26:16,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:16,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:26:16,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:26:49,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:49,877 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:26:49,882 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:26:49,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:49,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 248, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:26:49,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:49,888 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 248, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:26:53,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:26:53,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:26:53,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.86 seconds 2025-02-15 09:26:53,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:53,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14696.81 MB 2025-02-15 09:26:53,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15574.47 MB 2025-02-15 09:26:53,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 877.66 MB 2025-02-15 09:26:53,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53865.35 MB 2025-02-15 09:26:53,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 09:26:53,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35500.59 MB 2025-02-15 09:26:53,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24394.67 MB 2025-02-15 09:26:53,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:26:53,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:26:53,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:26:53,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:53,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15574.47 MB 2025-02-15 09:26:53,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.77 MB 2025-02-15 09:26:53,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 165.31 MB 2025-02-15 09:26:53,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 09:26:53,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19862.13 MB 2025-02-15 09:26:53,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1497.37 MB 2025-02-15 09:26:53,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18538.18 MB 2025-02-15 09:26:54,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:26:54,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:26:54,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-15 09:26:54,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:54,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15739.77 MB 2025-02-15 09:26:54,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16019.79 MB 2025-02-15 09:26:54,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 280.02 MB 2025-02-15 09:26:54,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19862.13 MB 2025-02-15 09:26:54,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19113.44 MB 2025-02-15 09:26:54,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -748.68 MB 2025-02-15 09:26:54,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19995.40 MB 2025-02-15 09:26:54,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:26:54,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:26:54,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:26:54,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:54,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16019.79 MB 2025-02-15 09:26:54,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17016.28 MB 2025-02-15 09:26:54,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 996.49 MB 2025-02-15 09:26:54,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19113.44 MB 2025-02-15 09:26:54,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19612.57 MB 2025-02-15 09:26:54,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 499.12 MB 2025-02-15 09:26:54,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17763.98 MB 2025-02-15 09:26:54,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:26:54,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:26:54,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 09:26:54,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:54,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17016.28 MB 2025-02-15 09:26:54,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18198.89 MB 2025-02-15 09:26:54,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1182.61 MB 2025-02-15 09:26:54,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19612.57 MB 2025-02-15 09:26:54,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22607.30 MB 2025-02-15 09:26:54,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2994.73 MB 2025-02-15 09:26:54,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21124.78 MB 2025-02-15 09:26:54,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:26:54,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:26:54,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:26:54,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:54,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16019.79 MB 2025-02-15 09:26:54,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18198.89 MB 2025-02-15 09:26:54,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2179.10 MB 2025-02-15 09:26:54,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19113.44 MB 2025-02-15 09:26:54,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22607.30 MB 2025-02-15 09:26:54,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3493.86 MB 2025-02-15 09:26:54,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21124.78 MB 2025-02-15 09:26:55,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:26:55,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:26:55,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:26:55,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:55,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19007.83 MB 2025-02-15 09:26:55,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19413.74 MB 2025-02-15 09:26:55,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 405.90 MB 2025-02-15 09:26:55,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22607.30 MB 2025-02-15 09:26:55,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22827.50 MB 2025-02-15 09:26:55,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 220.20 MB 2025-02-15 09:26:55,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19789.92 MB 2025-02-15 09:26:55,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:26:55,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:26:55,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:26:55,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:55,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19631.54 MB 2025-02-15 09:26:55,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19840.30 MB 2025-02-15 09:26:55,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.76 MB 2025-02-15 09:26:55,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22827.50 MB 2025-02-15 09:26:55,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22827.50 MB 2025-02-15 09:26:55,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:26:55,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19879.25 MB 2025-02-15 09:26:55,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:26:55,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:26:55,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.16 seconds 2025-02-15 09:26:55,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:55,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13832.76 MB 2025-02-15 09:26:55,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20041.35 MB 2025-02-15 09:26:55,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6208.59 MB 2025-02-15 09:26:55,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53865.35 MB 2025-02-15 09:26:55,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22827.50 MB 2025-02-15 09:26:55,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31037.85 MB 2025-02-15 09:26:55,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20041.35 MB 2025-02-15 09:26:55,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:26:55,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:26:55,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:26:55,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:55,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14932.34 MB 2025-02-15 09:26:55,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17946.00 MB 2025-02-15 09:26:55,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3013.66 MB 2025-02-15 09:26:55,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22827.50 MB 2025-02-15 09:26:55,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22827.50 MB 2025-02-15 09:26:55,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:26:55,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18247.33 MB 2025-02-15 09:26:55,338 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 09:26:55,339 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:26:55,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:26:55,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:26:55,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:26:55,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:26:55,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17946.00 MB 2025-02-15 09:26:55,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26384.84 MB 2025-02-15 09:26:55,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 09:26:55,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22827.50 MB 2025-02-15 09:26:55,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33313.26 MB 2025-02-15 09:26:55,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 09:26:55,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26384.84 MB 2025-02-15 09:26:55,502 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 09:26:55,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:55,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:26:55,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:55,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:26:55,509 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:26:55,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:26:55,510 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:26:55,510 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:27:05,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:05,110 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:27:05,115 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:27:05,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:05,118 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 696, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:27:05,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:05,119 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 696, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:27:15,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:27:15,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:27:15,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.80 seconds 2025-02-15 09:27:15,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:15,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17818.55 MB 2025-02-15 09:27:15,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20282.70 MB 2025-02-15 09:27:15,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2464.15 MB 2025-02-15 09:27:15,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41701.87 MB 2025-02-15 09:27:15,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25230.84 MB 2025-02-15 09:27:15,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16471.03 MB 2025-02-15 09:27:15,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29101.86 MB 2025-02-15 09:27:15,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:27:15,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:27:15,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:27:15,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:15,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20282.70 MB 2025-02-15 09:27:15,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19396.14 MB 2025-02-15 09:27:15,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -886.56 MB 2025-02-15 09:27:15,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25230.84 MB 2025-02-15 09:27:15,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31241.27 MB 2025-02-15 09:27:15,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6010.44 MB 2025-02-15 09:27:15,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29102.35 MB 2025-02-15 09:27:17,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:27:17,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:27:17,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:27:17,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:17,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19396.14 MB 2025-02-15 09:27:17,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19926.98 MB 2025-02-15 09:27:17,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:27:17,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31241.27 MB 2025-02-15 09:27:17,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24182.26 MB 2025-02-15 09:27:17,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7059.01 MB 2025-02-15 09:27:17,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23906.57 MB 2025-02-15 09:27:17,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:27:17,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:27:17,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:27:17,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:17,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19926.98 MB 2025-02-15 09:27:17,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21816.51 MB 2025-02-15 09:27:17,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:27:17,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24182.26 MB 2025-02-15 09:27:17,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26069.70 MB 2025-02-15 09:27:17,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:27:17,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23233.94 MB 2025-02-15 09:27:18,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:27:18,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:27:18,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:27:18,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21816.51 MB 2025-02-15 09:27:18,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24058.37 MB 2025-02-15 09:27:18,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:27:18,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26069.70 MB 2025-02-15 09:27:18,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31734.10 MB 2025-02-15 09:27:18,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 09:27:18,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29602.65 MB 2025-02-15 09:27:18,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:27:18,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:27:18,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:27:18,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19926.98 MB 2025-02-15 09:27:18,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24058.37 MB 2025-02-15 09:27:18,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:27:18,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24182.26 MB 2025-02-15 09:27:18,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31734.10 MB 2025-02-15 09:27:18,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-15 09:27:18,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29602.65 MB 2025-02-15 09:27:18,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:27:18,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:27:18,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:27:18,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25591.91 MB 2025-02-15 09:27:18,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26358.91 MB 2025-02-15 09:27:18,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:27:18,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31734.10 MB 2025-02-15 09:27:18,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-15 09:27:18,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:27:18,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27066.70 MB 2025-02-15 09:27:18,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:27:18,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:27:18,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:27:18,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26771.80 MB 2025-02-15 09:27:18,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26999.10 MB 2025-02-15 09:27:18,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.30 MB 2025-02-15 09:27:18,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-15 09:27:18,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-15 09:27:18,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:27:18,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27174.80 MB 2025-02-15 09:27:18,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:27:18,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:27:18,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.20 seconds 2025-02-15 09:27:18,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.63 MB 2025-02-15 09:27:18,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27199.96 MB 2025-02-15 09:27:18,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11806.33 MB 2025-02-15 09:27:18,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41701.87 MB 2025-02-15 09:27:18,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-15 09:27:18,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9550.43 MB 2025-02-15 09:27:18,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27199.96 MB 2025-02-15 09:27:18,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:27:18,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:27:18,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:27:18,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27199.96 MB 2025-02-15 09:27:18,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20383.54 MB 2025-02-15 09:27:18,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6816.42 MB 2025-02-15 09:27:18,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-15 09:27:18,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-15 09:27:18,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:27:18,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29699.34 MB 2025-02-15 09:27:18,614 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 09:27:18,614 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:27:18,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:27:18,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:27:18,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:27:18,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:27:18,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20383.54 MB 2025-02-15 09:27:18,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28780.94 MB 2025-02-15 09:27:18,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 09:27:18,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-15 09:27:18,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40502.30 MB 2025-02-15 09:27:18,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 09:27:18,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28780.94 MB 2025-02-15 09:27:18,780 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 09:27:18,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:18,782 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:27:18,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:18,783 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:27:18,788 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:27:18,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:27:18,789 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:27:18,789 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:28:13,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:13,723 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:28:13,728 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:28:13,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:13,732 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 173, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:28:13,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:13,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 173, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:28:16,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:28:16,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:28:16,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-15 09:28:16,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:16,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14174.20 MB 2025-02-15 09:28:16,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14786.44 MB 2025-02-15 09:28:16,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 612.24 MB 2025-02-15 09:28:16,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48853.16 MB 2025-02-15 09:28:16,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:16,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26765.95 MB 2025-02-15 09:28:16,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23645.57 MB 2025-02-15 09:28:16,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:28:16,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:28:16,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:28:16,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:16,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14786.44 MB 2025-02-15 09:28:16,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15083.06 MB 2025-02-15 09:28:16,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 296.63 MB 2025-02-15 09:28:16,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:16,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:16,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:16,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17216.46 MB 2025-02-15 09:28:17,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:28:17,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:28:17,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 09:28:17,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15083.06 MB 2025-02-15 09:28:17,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15312.65 MB 2025-02-15 09:28:17,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.59 MB 2025-02-15 09:28:17,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:17,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:17,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:17,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19252.71 MB 2025-02-15 09:28:17,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:28:17,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:28:17,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:28:17,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-15 09:28:17,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16129.61 MB 2025-02-15 09:28:17,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.03 MB 2025-02-15 09:28:17,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:17,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:17,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:17,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16742.65 MB 2025-02-15 09:28:17,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:28:17,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:28:17,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:28:17,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16129.61 MB 2025-02-15 09:28:17,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17099.25 MB 2025-02-15 09:28:17,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 969.64 MB 2025-02-15 09:28:17,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:17,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:17,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:17,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19497.12 MB 2025-02-15 09:28:17,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:28:17,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:28:17,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:28:17,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-15 09:28:17,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17099.25 MB 2025-02-15 09:28:17,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1786.66 MB 2025-02-15 09:28:17,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:17,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 09:28:17,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:17,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19497.12 MB 2025-02-15 09:28:17,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:28:17,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:28:17,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 09:28:17,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.51 MB 2025-02-15 09:28:17,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18094.24 MB 2025-02-15 09:28:17,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 331.73 MB 2025-02-15 09:28:17,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 09:28:17,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22265.46 MB 2025-02-15 09:28:17,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-15 09:28:17,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18407.05 MB 2025-02-15 09:28:17,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:28:17,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:28:17,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:28:17,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18272.82 MB 2025-02-15 09:28:17,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18486.53 MB 2025-02-15 09:28:17,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.71 MB 2025-02-15 09:28:17,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22265.46 MB 2025-02-15 09:28:17,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22267.56 MB 2025-02-15 09:28:17,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 09:28:17,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18512.62 MB 2025-02-15 09:28:17,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:28:17,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:28:17,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.69 seconds 2025-02-15 09:28:17,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13571.45 MB 2025-02-15 09:28:17,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18687.60 MB 2025-02-15 09:28:17,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5116.15 MB 2025-02-15 09:28:17,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48853.16 MB 2025-02-15 09:28:17,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22267.56 MB 2025-02-15 09:28:17,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26585.60 MB 2025-02-15 09:28:17,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18687.60 MB 2025-02-15 09:28:17,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:28:17,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:28:17,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 09:28:17,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18687.60 MB 2025-02-15 09:28:17,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17504.56 MB 2025-02-15 09:28:17,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1183.04 MB 2025-02-15 09:28:17,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22267.56 MB 2025-02-15 09:28:17,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22267.56 MB 2025-02-15 09:28:17,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:28:17,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18922.69 MB 2025-02-15 09:28:17,712 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:28:17,712 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:28:17,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:28:17,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:28:17,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:28:17,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:28:17,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17504.56 MB 2025-02-15 09:28:17,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25943.58 MB 2025-02-15 09:28:17,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:28:17,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22267.56 MB 2025-02-15 09:28:17,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30658.27 MB 2025-02-15 09:28:17,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:28:17,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25943.58 MB 2025-02-15 09:28:17,881 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:28:17,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:17,882 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:28:17,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:17,883 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:28:17,888 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:28:17,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:28:17,889 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:28:17,889 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:29:05,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:05,635 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:29:05,640 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:29:05,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:05,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:29:05,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:05,644 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:29:24,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:29:24,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:29:24,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.90 seconds 2025-02-15 09:29:24,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:24,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21546.51 MB 2025-02-15 09:29:24,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25902.95 MB 2025-02-15 09:29:24,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4356.44 MB 2025-02-15 09:29:24,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43243.27 MB 2025-02-15 09:29:24,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33416.02 MB 2025-02-15 09:29:24,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9827.25 MB 2025-02-15 09:29:24,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34868.25 MB 2025-02-15 09:29:24,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:29:24,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:29:24,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:29:24,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:24,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25902.95 MB 2025-02-15 09:29:24,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22177.44 MB 2025-02-15 09:29:24,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3725.52 MB 2025-02-15 09:29:24,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33416.02 MB 2025-02-15 09:29:24,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41945.14 MB 2025-02-15 09:29:24,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8529.12 MB 2025-02-15 09:29:24,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38726.94 MB 2025-02-15 09:29:26,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:29:26,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:29:26,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 09:29:26,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22177.44 MB 2025-02-15 09:29:26,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22708.28 MB 2025-02-15 09:29:26,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:29:26,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41945.14 MB 2025-02-15 09:29:26,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 09:29:26,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17081.30 MB 2025-02-15 09:29:26,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26687.86 MB 2025-02-15 09:29:26,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:29:26,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:29:26,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:29:26,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-15 09:29:26,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24597.81 MB 2025-02-15 09:29:26,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:29:26,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 09:29:26,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27694.99 MB 2025-02-15 09:29:26,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:29:26,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26015.24 MB 2025-02-15 09:29:26,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:29:26,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:29:26,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:29:26,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24597.81 MB 2025-02-15 09:29:26,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-15 09:29:26,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:29:26,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27694.99 MB 2025-02-15 09:29:26,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34303.12 MB 2025-02-15 09:29:26,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 09:29:26,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-15 09:29:26,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:29:26,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:29:26,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:29:26,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-15 09:29:26,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-15 09:29:26,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:29:26,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 09:29:26,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34303.12 MB 2025-02-15 09:29:26,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-15 09:29:26,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-15 09:29:26,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:29:26,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:29:26,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:29:26,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.21 MB 2025-02-15 09:29:26,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29140.21 MB 2025-02-15 09:29:26,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:29:26,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34303.12 MB 2025-02-15 09:29:26,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 09:29:26,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:29:26,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29848.00 MB 2025-02-15 09:29:26,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:29:26,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:29:26,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:29:26,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29553.10 MB 2025-02-15 09:29:26,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29781.99 MB 2025-02-15 09:29:26,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-15 09:29:26,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34720.45 MB 2025-02-15 09:29:26,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 09:29:26,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:29:26,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30021.02 MB 2025-02-15 09:29:26,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:29:26,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:29:26,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.34 seconds 2025-02-15 09:29:26,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:26,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17257.61 MB 2025-02-15 09:29:26,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29982.84 MB 2025-02-15 09:29:26,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12725.23 MB 2025-02-15 09:29:26,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43243.27 MB 2025-02-15 09:29:26,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 09:29:26,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8522.83 MB 2025-02-15 09:29:26,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30021.02 MB 2025-02-15 09:29:27,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:29:27,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:29:27,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:29:27,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:27,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29982.84 MB 2025-02-15 09:29:27,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22254.21 MB 2025-02-15 09:29:27,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7728.63 MB 2025-02-15 09:29:27,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34720.45 MB 2025-02-15 09:29:27,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-15 09:29:27,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:29:27,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32487.75 MB 2025-02-15 09:29:27,276 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 09:29:27,276 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:29:27,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:29:27,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:29:27,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:29:27,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:29:27,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22254.21 MB 2025-02-15 09:29:27,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30670.81 MB 2025-02-15 09:29:27,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 09:29:27,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34720.45 MB 2025-02-15 09:29:27,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43088.08 MB 2025-02-15 09:29:27,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 09:29:27,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30670.81 MB 2025-02-15 09:29:27,442 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 09:29:27,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:27,443 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:29:27,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:27,444 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:29:27,449 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:29:27,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:29:27,450 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:29:27,450 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:30:16,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:16,081 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:30:16,086 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:30:16,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:16,090 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1088, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:30:16,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:16,091 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1088, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:30:32,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:30:32,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:30:32,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.86 seconds 2025-02-15 09:30:32,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:32,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20550.06 MB 2025-02-15 09:30:32,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24400.44 MB 2025-02-15 09:30:32,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3850.37 MB 2025-02-15 09:30:32,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51455.72 MB 2025-02-15 09:30:32,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28705.82 MB 2025-02-15 09:30:32,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22749.90 MB 2025-02-15 09:30:32,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33418.82 MB 2025-02-15 09:30:33,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:30:33,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:30:33,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:30:33,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:33,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24400.44 MB 2025-02-15 09:30:33,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21435.07 MB 2025-02-15 09:30:33,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2965.36 MB 2025-02-15 09:30:33,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28705.82 MB 2025-02-15 09:30:33,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37771.80 MB 2025-02-15 09:30:33,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9065.99 MB 2025-02-15 09:30:33,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35566.27 MB 2025-02-15 09:30:34,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:30:34,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:30:34,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:30:34,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:34,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21435.07 MB 2025-02-15 09:30:34,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21965.91 MB 2025-02-15 09:30:34,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:30:34,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37771.80 MB 2025-02-15 09:30:34,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26979.86 MB 2025-02-15 09:30:34,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10791.94 MB 2025-02-15 09:30:34,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25944.46 MB 2025-02-15 09:30:34,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:30:34,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:30:34,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:30:34,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:34,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21965.91 MB 2025-02-15 09:30:34,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23855.45 MB 2025-02-15 09:30:34,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:30:34,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26979.86 MB 2025-02-15 09:30:34,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27923.58 MB 2025-02-15 09:30:34,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:30:34,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25272.88 MB 2025-02-15 09:30:35,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:30:35,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:30:35,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:30:35,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23855.45 MB 2025-02-15 09:30:35,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26097.30 MB 2025-02-15 09:30:35,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:30:35,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27923.58 MB 2025-02-15 09:30:35,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33585.89 MB 2025-02-15 09:30:35,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:30:35,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31641.58 MB 2025-02-15 09:30:35,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:30:35,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:30:35,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:30:35,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21965.91 MB 2025-02-15 09:30:35,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26097.30 MB 2025-02-15 09:30:35,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:30:35,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26979.86 MB 2025-02-15 09:30:35,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33585.89 MB 2025-02-15 09:30:35,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:30:35,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31641.58 MB 2025-02-15 09:30:35,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:30:35,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:30:35,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:30:35,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27630.85 MB 2025-02-15 09:30:35,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28397.85 MB 2025-02-15 09:30:35,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:30:35,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33585.89 MB 2025-02-15 09:30:35,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-15 09:30:35,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:30:35,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29105.64 MB 2025-02-15 09:30:35,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:30:35,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:30:35,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:30:35,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28810.74 MB 2025-02-15 09:30:35,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29039.80 MB 2025-02-15 09:30:35,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 09:30:35,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34003.22 MB 2025-02-15 09:30:35,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-15 09:30:35,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:30:35,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29234.66 MB 2025-02-15 09:30:35,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:30:35,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:30:35,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.29 seconds 2025-02-15 09:30:35,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16759.39 MB 2025-02-15 09:30:35,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29240.77 MB 2025-02-15 09:30:35,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12481.38 MB 2025-02-15 09:30:35,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51455.72 MB 2025-02-15 09:30:35,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-15 09:30:35,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17452.50 MB 2025-02-15 09:30:35,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29240.77 MB 2025-02-15 09:30:35,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:30:35,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:30:35,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:30:35,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29240.77 MB 2025-02-15 09:30:35,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21762.25 MB 2025-02-15 09:30:35,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7478.52 MB 2025-02-15 09:30:35,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34003.22 MB 2025-02-15 09:30:35,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-15 09:30:35,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:30:35,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31751.21 MB 2025-02-15 09:30:35,673 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 09:30:35,674 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:30:35,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:30:35,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:30:35,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:30:35,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:30:35,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21762.25 MB 2025-02-15 09:30:35,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30197.10 MB 2025-02-15 09:30:35,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 09:30:35,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34003.22 MB 2025-02-15 09:30:35,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42389.73 MB 2025-02-15 09:30:35,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 09:30:35,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30197.10 MB 2025-02-15 09:30:35,840 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 09:30:35,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:35,841 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:30:35,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:35,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:30:35,847 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:30:35,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:30:35,848 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:30:35,848 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:31:53,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:31:53,520 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:31:53,529 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:31:53,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:31:53,534 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:31:53,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:31:53,535 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:32:10,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:32:10,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:32:10,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.08 seconds 2025-02-15 09:32:10,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:10,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20661.56 MB 2025-02-15 09:32:10,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24568.55 MB 2025-02-15 09:32:10,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-15 09:32:10,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54968.45 MB 2025-02-15 09:32:10,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28762.44 MB 2025-02-15 09:32:10,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26206.01 MB 2025-02-15 09:32:10,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33530.31 MB 2025-02-15 09:32:10,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:32:10,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:32:10,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:32:10,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:10,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24568.55 MB 2025-02-15 09:32:10,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21518.25 MB 2025-02-15 09:32:10,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3050.30 MB 2025-02-15 09:32:10,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28762.44 MB 2025-02-15 09:32:10,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38012.98 MB 2025-02-15 09:32:10,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9250.54 MB 2025-02-15 09:32:10,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35916.65 MB 2025-02-15 09:32:12,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:32:12,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:32:12,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 09:32:12,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:12,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21518.25 MB 2025-02-15 09:32:12,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22049.09 MB 2025-02-15 09:32:12,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:32:12,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38012.98 MB 2025-02-15 09:32:12,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26979.86 MB 2025-02-15 09:32:12,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11033.12 MB 2025-02-15 09:32:12,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26027.64 MB 2025-02-15 09:32:12,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:32:12,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:32:12,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:32:12,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:12,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22049.09 MB 2025-02-15 09:32:12,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23938.63 MB 2025-02-15 09:32:12,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:32:12,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26979.86 MB 2025-02-15 09:32:12,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27923.58 MB 2025-02-15 09:32:12,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:32:12,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25356.06 MB 2025-02-15 09:32:12,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:32:12,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:32:12,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:32:12,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:12,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23938.63 MB 2025-02-15 09:32:12,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26180.48 MB 2025-02-15 09:32:12,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:32:12,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27923.58 MB 2025-02-15 09:32:12,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33585.89 MB 2025-02-15 09:32:12,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:32:12,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31724.76 MB 2025-02-15 09:32:12,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:32:12,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:32:12,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:32:12,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:12,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22049.09 MB 2025-02-15 09:32:12,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26180.48 MB 2025-02-15 09:32:12,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:32:12,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26979.86 MB 2025-02-15 09:32:12,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33585.89 MB 2025-02-15 09:32:12,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:32:12,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31724.76 MB 2025-02-15 09:32:13,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:32:13,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:32:13,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:32:13,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:13,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27714.02 MB 2025-02-15 09:32:13,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28481.03 MB 2025-02-15 09:32:13,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:32:13,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33585.89 MB 2025-02-15 09:32:13,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-15 09:32:13,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:32:13,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29188.82 MB 2025-02-15 09:32:13,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:32:13,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:32:13,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:32:13,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:13,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28893.92 MB 2025-02-15 09:32:13,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29123.12 MB 2025-02-15 09:32:13,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-15 09:32:13,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34001.13 MB 2025-02-15 09:32:13,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-15 09:32:13,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:32:13,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29331.49 MB 2025-02-15 09:32:13,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:32:13,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:32:13,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.50 seconds 2025-02-15 09:32:13,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:13,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16815.13 MB 2025-02-15 09:32:13,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29324.10 MB 2025-02-15 09:32:13,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12508.97 MB 2025-02-15 09:32:13,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54968.45 MB 2025-02-15 09:32:13,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-15 09:32:13,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20967.33 MB 2025-02-15 09:32:13,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29331.49 MB 2025-02-15 09:32:13,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:32:13,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:32:13,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:32:13,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:13,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29324.10 MB 2025-02-15 09:32:13,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21818.00 MB 2025-02-15 09:32:13,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7506.10 MB 2025-02-15 09:32:13,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34001.13 MB 2025-02-15 09:32:13,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-15 09:32:13,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:32:13,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31834.54 MB 2025-02-15 09:32:13,326 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 09:32:13,326 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:32:13,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:32:13,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:32:13,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:32:13,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:32:13,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21818.00 MB 2025-02-15 09:32:13,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30252.85 MB 2025-02-15 09:32:13,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 09:32:13,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34001.13 MB 2025-02-15 09:32:13,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42387.64 MB 2025-02-15 09:32:13,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 09:32:13,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30252.85 MB 2025-02-15 09:32:13,495 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 09:32:13,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:32:13,497 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:32:13,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:32:13,498 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:32:13,502 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:32:13,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:32:13,504 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:32:13,504 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:33:37,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:33:37,536 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:33:37,541 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:33:37,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:33:37,545 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1973, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:33:37,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:33:37,546 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1973, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:34:07,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:34:07,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:34:07,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.36 seconds 2025-02-15 09:34:07,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:07,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26716.89 MB 2025-02-15 09:34:07,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33699.22 MB 2025-02-15 09:34:07,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6982.34 MB 2025-02-15 09:34:07,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54966.35 MB 2025-02-15 09:34:07,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-15 09:34:07,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14745.08 MB 2025-02-15 09:34:07,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42530.04 MB 2025-02-15 09:34:08,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:34:08,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:34:08,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:34:08,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:08,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33699.22 MB 2025-02-15 09:34:08,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26034.86 MB 2025-02-15 09:34:08,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7664.36 MB 2025-02-15 09:34:08,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40221.28 MB 2025-02-15 09:34:08,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55306.09 MB 2025-02-15 09:34:08,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15084.81 MB 2025-02-15 09:34:08,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54254.01 MB 2025-02-15 09:34:10,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:34:10,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:34:10,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:34:10,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26034.86 MB 2025-02-15 09:34:10,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26565.70 MB 2025-02-15 09:34:10,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:34:10,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-15 09:34:10,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30469.52 MB 2025-02-15 09:34:10,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24836.57 MB 2025-02-15 09:34:10,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30545.29 MB 2025-02-15 09:34:10,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:34:10,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:34:10,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:34:10,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.70 MB 2025-02-15 09:34:10,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28455.24 MB 2025-02-15 09:34:10,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:34:10,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30469.52 MB 2025-02-15 09:34:10,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-15 09:34:10,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:34:10,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29872.67 MB 2025-02-15 09:34:10,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:34:10,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:34:10,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:34:10,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28455.24 MB 2025-02-15 09:34:10,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30697.09 MB 2025-02-15 09:34:10,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:34:10,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32356.96 MB 2025-02-15 09:34:10,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38019.27 MB 2025-02-15 09:34:10,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:34:10,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36241.38 MB 2025-02-15 09:34:10,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:34:10,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:34:10,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:34:10,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.70 MB 2025-02-15 09:34:10,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30697.09 MB 2025-02-15 09:34:10,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:34:10,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30469.52 MB 2025-02-15 09:34:10,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38019.27 MB 2025-02-15 09:34:10,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 09:34:10,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36241.38 MB 2025-02-15 09:34:10,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:34:10,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:34:10,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:34:10,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32230.64 MB 2025-02-15 09:34:10,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32997.64 MB 2025-02-15 09:34:10,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:34:10,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38019.27 MB 2025-02-15 09:34:10,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38436.60 MB 2025-02-15 09:34:10,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:34:10,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33705.43 MB 2025-02-15 09:34:10,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:34:10,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:34:10,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:34:10,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33410.53 MB 2025-02-15 09:34:10,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33639.98 MB 2025-02-15 09:34:10,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.45 MB 2025-02-15 09:34:10,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38436.60 MB 2025-02-15 09:34:10,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38436.60 MB 2025-02-15 09:34:10,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:34:10,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33858.74 MB 2025-02-15 09:34:10,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:34:10,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:34:10,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.90 seconds 2025-02-15 09:34:10,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19842.80 MB 2025-02-15 09:34:10,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33841.05 MB 2025-02-15 09:34:10,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13998.26 MB 2025-02-15 09:34:10,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54966.35 MB 2025-02-15 09:34:10,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38436.60 MB 2025-02-15 09:34:10,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16529.75 MB 2025-02-15 09:34:10,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33858.74 MB 2025-02-15 09:34:10,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:34:10,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:34:10,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:34:10,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33841.05 MB 2025-02-15 09:34:10,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24848.22 MB 2025-02-15 09:34:10,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8992.84 MB 2025-02-15 09:34:10,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38436.60 MB 2025-02-15 09:34:10,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38436.60 MB 2025-02-15 09:34:10,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:34:10,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36352.72 MB 2025-02-15 09:34:10,734 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:34:10,734 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:34:10,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:34:10,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:34:10,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:34:10,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:34:10,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24848.22 MB 2025-02-15 09:34:10,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33287.24 MB 2025-02-15 09:34:10,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:34:10,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38436.60 MB 2025-02-15 09:34:10,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46827.31 MB 2025-02-15 09:34:10,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:34:10,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33287.24 MB 2025-02-15 09:34:10,905 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:34:10,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:34:10,906 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:34:10,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:34:10,907 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:34:10,912 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:34:10,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:34:10,913 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:34:10,913 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:35:00,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:00,887 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:35:00,896 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:35:00,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:00,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2330, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:35:00,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:00,907 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2330, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:35:37,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:35:37,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:35:37,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.26 seconds 2025-02-15 09:35:37,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:37,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29204.52 MB 2025-02-15 09:35:37,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37450.52 MB 2025-02-15 09:35:37,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8246.00 MB 2025-02-15 09:35:37,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59412.32 MB 2025-02-15 09:35:37,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41498.44 MB 2025-02-15 09:35:37,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17913.87 MB 2025-02-15 09:35:37,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46376.63 MB 2025-02-15 09:35:37,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:35:37,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:35:37,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:35:37,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:37,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37450.52 MB 2025-02-15 09:35:37,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27891.84 MB 2025-02-15 09:35:37,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9558.68 MB 2025-02-15 09:35:37,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41498.44 MB 2025-02-15 09:35:37,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53550.78 MB 2025-02-15 09:35:37,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12052.33 MB 2025-02-15 09:35:37,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52387.31 MB 2025-02-15 09:35:39,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:35:39,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:35:39,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:35:39,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27891.84 MB 2025-02-15 09:35:39,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28422.68 MB 2025-02-15 09:35:39,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:35:39,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53550.78 MB 2025-02-15 09:35:39,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31184.65 MB 2025-02-15 09:35:39,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22366.13 MB 2025-02-15 09:35:39,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32401.23 MB 2025-02-15 09:35:39,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:35:39,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:35:39,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:35:39,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28422.68 MB 2025-02-15 09:35:39,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30312.22 MB 2025-02-15 09:35:39,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:35:39,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31184.65 MB 2025-02-15 09:35:39,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34015.81 MB 2025-02-15 09:35:39,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:35:39,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31729.65 MB 2025-02-15 09:35:39,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:35:39,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:35:39,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:35:39,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.22 MB 2025-02-15 09:35:39,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32554.07 MB 2025-02-15 09:35:39,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:35:39,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34015.81 MB 2025-02-15 09:35:39,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-15 09:35:39,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:35:39,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38098.36 MB 2025-02-15 09:35:39,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:35:39,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:35:39,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:35:39,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28422.68 MB 2025-02-15 09:35:39,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32554.07 MB 2025-02-15 09:35:39,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:35:39,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31184.65 MB 2025-02-15 09:35:39,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-15 09:35:39,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:35:39,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38098.36 MB 2025-02-15 09:35:39,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:35:39,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:35:39,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:35:39,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34087.62 MB 2025-02-15 09:35:39,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34854.62 MB 2025-02-15 09:35:39,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:35:39,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40149.98 MB 2025-02-15 09:35:39,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-15 09:35:39,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:35:39,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35562.41 MB 2025-02-15 09:35:39,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:35:39,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:35:39,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:35:39,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35267.51 MB 2025-02-15 09:35:39,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35496.68 MB 2025-02-15 09:35:39,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-15 09:35:39,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40565.21 MB 2025-02-15 09:35:39,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-15 09:35:39,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:35:39,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35731.07 MB 2025-02-15 09:35:39,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:35:39,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:35:39,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.81 seconds 2025-02-15 09:35:39,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21086.61 MB 2025-02-15 09:35:39,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.75 MB 2025-02-15 09:35:39,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14611.14 MB 2025-02-15 09:35:39,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59412.32 MB 2025-02-15 09:35:39,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-15 09:35:39,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18847.11 MB 2025-02-15 09:35:39,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35731.07 MB 2025-02-15 09:35:39,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:35:39,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:35:39,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:35:39,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:39,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35697.75 MB 2025-02-15 09:35:39,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26091.00 MB 2025-02-15 09:35:39,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9606.75 MB 2025-02-15 09:35:39,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40565.21 MB 2025-02-15 09:35:39,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-15 09:35:39,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:35:39,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38209.42 MB 2025-02-15 09:35:40,010 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:35:40,010 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:35:40,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:35:40,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:35:40,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:35:40,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:35:40,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26091.00 MB 2025-02-15 09:35:40,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34530.02 MB 2025-02-15 09:35:40,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:35:40,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40565.21 MB 2025-02-15 09:35:40,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48955.92 MB 2025-02-15 09:35:40,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:35:40,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34530.02 MB 2025-02-15 09:35:40,184 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:35:40,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:40,185 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:35:40,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:40,186 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:35:40,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:35:40,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:35:40,192 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:35:40,192 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:36:41,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:41,218 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:36:41,223 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:36:41,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:41,227 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:36:41,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:41,228 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:36:56,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:36:56,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:36:56,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.53 seconds 2025-02-15 09:36:56,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:56,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19964.74 MB 2025-02-15 09:36:56,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23517.84 MB 2025-02-15 09:36:56,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3553.10 MB 2025-02-15 09:36:56,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61540.93 MB 2025-02-15 09:36:56,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28422.70 MB 2025-02-15 09:36:56,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33118.22 MB 2025-02-15 09:36:56,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32380.51 MB 2025-02-15 09:36:56,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:36:56,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:36:56,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:36:56,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:56,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23517.84 MB 2025-02-15 09:36:56,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20997.33 MB 2025-02-15 09:36:56,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2520.51 MB 2025-02-15 09:36:56,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28422.70 MB 2025-02-15 09:36:56,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36893.10 MB 2025-02-15 09:36:56,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8470.40 MB 2025-02-15 09:36:56,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34794.04 MB 2025-02-15 09:36:58,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:36:58,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:36:58,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:36:58,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:58,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20997.33 MB 2025-02-15 09:36:58,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21528.18 MB 2025-02-15 09:36:58,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:36:58,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36893.10 MB 2025-02-15 09:36:58,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-15 09:36:58,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10609.49 MB 2025-02-15 09:36:58,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25506.72 MB 2025-02-15 09:36:58,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:36:58,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:36:58,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:36:58,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:58,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-15 09:36:58,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23417.71 MB 2025-02-15 09:36:58,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:36:58,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 09:36:58,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27227.32 MB 2025-02-15 09:36:58,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:36:58,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24835.14 MB 2025-02-15 09:36:58,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:36:58,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:36:58,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:36:58,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:58,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23417.71 MB 2025-02-15 09:36:58,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-15 09:36:58,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:36:58,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27227.32 MB 2025-02-15 09:36:58,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33833.35 MB 2025-02-15 09:36:58,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:36:58,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-15 09:36:58,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:36:58,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:36:58,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:36:58,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:58,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-15 09:36:58,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-15 09:36:58,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:36:58,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-15 09:36:58,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33833.35 MB 2025-02-15 09:36:58,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 09:36:58,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-15 09:36:59,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:36:59,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:36:59,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-15 09:36:59,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:59,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27193.11 MB 2025-02-15 09:36:59,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27960.11 MB 2025-02-15 09:36:59,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:36:59,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33833.35 MB 2025-02-15 09:36:59,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34246.49 MB 2025-02-15 09:36:59,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:36:59,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28667.90 MB 2025-02-15 09:36:59,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:36:59,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:36:59,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:36:59,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:59,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.00 MB 2025-02-15 09:36:59,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28601.38 MB 2025-02-15 09:36:59,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.38 MB 2025-02-15 09:36:59,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34246.49 MB 2025-02-15 09:36:59,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34246.49 MB 2025-02-15 09:36:59,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:36:59,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28800.99 MB 2025-02-15 09:36:59,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:36:59,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:36:59,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.13 seconds 2025-02-15 09:36:59,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:59,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16466.72 MB 2025-02-15 09:36:59,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28802.23 MB 2025-02-15 09:36:59,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12335.51 MB 2025-02-15 09:36:59,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61540.93 MB 2025-02-15 09:36:59,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34246.49 MB 2025-02-15 09:36:59,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27294.43 MB 2025-02-15 09:36:59,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28802.23 MB 2025-02-15 09:36:59,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:36:59,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:36:59,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:36:59,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:59,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28802.23 MB 2025-02-15 09:36:59,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31810.74 MB 2025-02-15 09:36:59,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-15 09:36:59,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34246.49 MB 2025-02-15 09:36:59,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34246.49 MB 2025-02-15 09:36:59,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:36:59,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32111.55 MB 2025-02-15 09:36:59,648 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 09:36:59,648 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:36:59,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:36:59,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:36:59,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:36:59,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:36:59,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31810.74 MB 2025-02-15 09:36:59,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40233.94 MB 2025-02-15 09:36:59,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 09:36:59,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34246.49 MB 2025-02-15 09:36:59,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44717.57 MB 2025-02-15 09:36:59,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 09:36:59,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40233.94 MB 2025-02-15 09:36:59,817 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 09:36:59,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:59,818 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:36:59,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:59,819 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:36:59,824 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:36:59,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:36:59,825 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:36:59,825 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:37:46,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:37:46,498 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:37:46,503 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:37:46,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:37:46,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1510, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:37:46,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:37:46,508 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1510, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:38:09,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:38:09,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:38:09,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.30 seconds 2025-02-15 09:38:09,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:09,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42910.32 MB 2025-02-15 09:38:09,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48254.13 MB 2025-02-15 09:38:09,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5343.81 MB 2025-02-15 09:38:09,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55685.68 MB 2025-02-15 09:38:09,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56103.01 MB 2025-02-15 09:38:09,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:38:09,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57138.84 MB 2025-02-15 09:38:09,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:38:09,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:38:09,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:38:09,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:09,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48254.13 MB 2025-02-15 09:38:09,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23628.79 MB 2025-02-15 09:38:09,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -24625.34 MB 2025-02-15 09:38:09,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56103.01 MB 2025-02-15 09:38:09,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41592.82 MB 2025-02-15 09:38:09,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14510.19 MB 2025-02-15 09:38:09,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48254.13 MB 2025-02-15 09:38:11,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:38:11,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:38:11,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:38:11,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:11,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23628.79 MB 2025-02-15 09:38:11,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24159.64 MB 2025-02-15 09:38:11,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:38:11,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41592.82 MB 2025-02-15 09:38:11,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27445.43 MB 2025-02-15 09:38:11,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14147.39 MB 2025-02-15 09:38:11,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28139.22 MB 2025-02-15 09:38:11,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:38:11,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:38:11,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:38:11,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:11,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.64 MB 2025-02-15 09:38:11,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26049.17 MB 2025-02-15 09:38:11,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:38:11,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27445.43 MB 2025-02-15 09:38:11,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29332.87 MB 2025-02-15 09:38:11,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:38:11,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27466.60 MB 2025-02-15 09:38:12,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:38:12,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:38:12,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:38:12,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26049.17 MB 2025-02-15 09:38:12,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28291.03 MB 2025-02-15 09:38:12,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:38:12,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29332.87 MB 2025-02-15 09:38:12,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-15 09:38:12,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:38:12,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33835.31 MB 2025-02-15 09:38:12,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:38:12,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:38:12,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:38:12,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.64 MB 2025-02-15 09:38:12,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28291.03 MB 2025-02-15 09:38:12,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:38:12,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27445.43 MB 2025-02-15 09:38:12,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-15 09:38:12,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 09:38:12,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33835.31 MB 2025-02-15 09:38:12,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:38:12,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:38:12,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:38:12,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29824.57 MB 2025-02-15 09:38:12,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30591.57 MB 2025-02-15 09:38:12,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:38:12,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35938.89 MB 2025-02-15 09:38:12,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 09:38:12,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:38:12,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31299.36 MB 2025-02-15 09:38:12,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:38:12,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:38:12,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:38:12,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31004.46 MB 2025-02-15 09:38:12,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31233.70 MB 2025-02-15 09:38:12,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.24 MB 2025-02-15 09:38:12,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36354.13 MB 2025-02-15 09:38:12,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 09:38:12,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:38:12,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31436.03 MB 2025-02-15 09:38:12,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:38:12,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:38:12,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.74 seconds 2025-02-15 09:38:12,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37648.44 MB 2025-02-15 09:38:12,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31434.78 MB 2025-02-15 09:38:12,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6213.66 MB 2025-02-15 09:38:12,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53095.69 MB 2025-02-15 09:38:12,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 09:38:12,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16741.56 MB 2025-02-15 09:38:12,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31436.03 MB 2025-02-15 09:38:12,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:38:12,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:38:12,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:38:12,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31434.78 MB 2025-02-15 09:38:12,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23234.06 MB 2025-02-15 09:38:12,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8200.72 MB 2025-02-15 09:38:12,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36354.13 MB 2025-02-15 09:38:12,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 09:38:12,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:38:12,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32439.44 MB 2025-02-15 09:38:12,536 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:38:12,536 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:38:12,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:38:12,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:38:12,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:38:12,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:38:12,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23234.06 MB 2025-02-15 09:38:12,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31673.08 MB 2025-02-15 09:38:12,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:38:12,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36354.13 MB 2025-02-15 09:38:12,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44744.84 MB 2025-02-15 09:38:12,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:38:12,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31673.08 MB 2025-02-15 09:38:12,703 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:38:12,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:12,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:38:12,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:12,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:38:12,711 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:38:12,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:12,712 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:38:12,712 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:38:49,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:49,432 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:38:49,438 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:38:49,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:49,442 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1048, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:38:49,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:38:49,443 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1048, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:39:05,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:39:05,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:39:05,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.33 seconds 2025-02-15 09:39:05,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:05,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20271.34 MB 2025-02-15 09:39:05,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23981.20 MB 2025-02-15 09:39:05,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3709.86 MB 2025-02-15 09:39:05,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57329.84 MB 2025-02-15 09:39:05,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26971.47 MB 2025-02-15 09:39:05,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30358.37 MB 2025-02-15 09:39:05,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32914.41 MB 2025-02-15 09:39:05,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:39:05,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:39:05,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:39:05,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:05,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23981.20 MB 2025-02-15 09:39:05,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21227.12 MB 2025-02-15 09:39:05,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2754.08 MB 2025-02-15 09:39:05,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-15 09:39:05,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34787.56 MB 2025-02-15 09:39:05,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7816.09 MB 2025-02-15 09:39:05,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33895.95 MB 2025-02-15 09:39:07,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:39:07,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:39:07,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:39:07,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:07,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21227.12 MB 2025-02-15 09:39:07,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21757.97 MB 2025-02-15 09:39:07,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:39:07,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34787.56 MB 2025-02-15 09:39:07,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25386.02 MB 2025-02-15 09:39:07,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9401.53 MB 2025-02-15 09:39:07,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25737.55 MB 2025-02-15 09:39:07,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:39:07,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:39:07,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:39:07,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:07,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21757.97 MB 2025-02-15 09:39:07,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23647.50 MB 2025-02-15 09:39:07,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:39:07,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25386.02 MB 2025-02-15 09:39:07,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27273.46 MB 2025-02-15 09:39:07,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:39:07,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25064.93 MB 2025-02-15 09:39:08,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:39:08,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:39:08,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:39:08,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23647.50 MB 2025-02-15 09:39:08,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25889.36 MB 2025-02-15 09:39:08,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:39:08,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27273.46 MB 2025-02-15 09:39:08,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32935.77 MB 2025-02-15 09:39:08,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:39:08,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31433.64 MB 2025-02-15 09:39:08,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:39:08,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:39:08,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:39:08,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21757.97 MB 2025-02-15 09:39:08,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25889.36 MB 2025-02-15 09:39:08,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:39:08,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25386.02 MB 2025-02-15 09:39:08,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32935.77 MB 2025-02-15 09:39:08,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 09:39:08,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31433.64 MB 2025-02-15 09:39:08,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:39:08,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:39:08,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:39:08,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27422.90 MB 2025-02-15 09:39:08,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28189.90 MB 2025-02-15 09:39:08,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:39:08,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32935.77 MB 2025-02-15 09:39:08,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33353.11 MB 2025-02-15 09:39:08,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:39:08,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28897.69 MB 2025-02-15 09:39:08,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:39:08,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:39:08,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:39:08,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28602.79 MB 2025-02-15 09:39:08,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28834.72 MB 2025-02-15 09:39:08,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.93 MB 2025-02-15 09:39:08,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33353.11 MB 2025-02-15 09:39:08,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33353.11 MB 2025-02-15 09:39:08,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:39:08,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29020.06 MB 2025-02-15 09:39:08,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:39:08,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:39:08,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.77 seconds 2025-02-15 09:39:08,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16620.02 MB 2025-02-15 09:39:08,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29035.80 MB 2025-02-15 09:39:08,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12415.77 MB 2025-02-15 09:39:08,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57329.84 MB 2025-02-15 09:39:08,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33353.11 MB 2025-02-15 09:39:08,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23976.74 MB 2025-02-15 09:39:08,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29035.80 MB 2025-02-15 09:39:08,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:39:08,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:39:08,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:39:08,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29035.80 MB 2025-02-15 09:39:08,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21624.41 MB 2025-02-15 09:39:08,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7411.39 MB 2025-02-15 09:39:08,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33353.11 MB 2025-02-15 09:39:08,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33353.11 MB 2025-02-15 09:39:08,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:39:08,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31547.46 MB 2025-02-15 09:39:08,501 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:39:08,501 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:39:08,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:39:08,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:39:08,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:39:08,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:39:08,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21624.41 MB 2025-02-15 09:39:08,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30063.43 MB 2025-02-15 09:39:08,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:39:08,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33353.11 MB 2025-02-15 09:39:08,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41743.81 MB 2025-02-15 09:39:08,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:39:08,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30063.43 MB 2025-02-15 09:39:08,670 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:39:08,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:39:08,671 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:39:08,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:39:08,672 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:39:08,677 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:39:08,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:39:08,678 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:39:08,678 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:40:04,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:04,197 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:40:04,202 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:40:04,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:04,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 646, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:40:04,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:04,207 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 646, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:40:14,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:40:14,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:40:14,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.98 seconds 2025-02-15 09:40:14,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:14,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17470.14 MB 2025-02-15 09:40:14,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19756.30 MB 2025-02-15 09:40:14,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2286.16 MB 2025-02-15 09:40:14,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54328.82 MB 2025-02-15 09:40:14,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22775.07 MB 2025-02-15 09:40:14,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31553.75 MB 2025-02-15 09:40:14,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28754.26 MB 2025-02-15 09:40:14,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:40:14,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:40:14,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 09:40:14,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:14,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19756.30 MB 2025-02-15 09:40:14,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19137.25 MB 2025-02-15 09:40:14,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -619.04 MB 2025-02-15 09:40:14,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22775.07 MB 2025-02-15 09:40:14,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29290.92 MB 2025-02-15 09:40:14,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6515.85 MB 2025-02-15 09:40:14,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28372.19 MB 2025-02-15 09:40:16,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:40:16,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:40:16,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:40:16,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19137.25 MB 2025-02-15 09:40:16,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.09 MB 2025-02-15 09:40:16,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:40:16,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29290.92 MB 2025-02-15 09:40:16,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22611.49 MB 2025-02-15 09:40:16,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6679.43 MB 2025-02-15 09:40:16,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23647.68 MB 2025-02-15 09:40:16,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:40:16,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:40:16,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:40:16,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-15 09:40:16,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21557.63 MB 2025-02-15 09:40:16,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:40:16,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22611.49 MB 2025-02-15 09:40:16,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24498.93 MB 2025-02-15 09:40:16,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:40:16,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22975.06 MB 2025-02-15 09:40:16,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:40:16,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:40:16,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:40:16,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21557.63 MB 2025-02-15 09:40:16,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-15 09:40:16,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:40:16,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24498.93 MB 2025-02-15 09:40:16,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31104.96 MB 2025-02-15 09:40:16,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:40:16,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-15 09:40:16,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:40:16,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:40:16,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:40:16,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-15 09:40:16,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-15 09:40:16,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:40:16,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22611.49 MB 2025-02-15 09:40:16,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31104.96 MB 2025-02-15 09:40:16,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 09:40:16,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-15 09:40:16,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:40:16,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:40:16,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:40:16,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.03 MB 2025-02-15 09:40:16,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.03 MB 2025-02-15 09:40:16,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:40:16,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31104.96 MB 2025-02-15 09:40:16,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-15 09:40:16,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:40:16,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26807.82 MB 2025-02-15 09:40:16,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:40:16,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:40:16,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:40:16,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26512.92 MB 2025-02-15 09:40:16,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.11 MB 2025-02-15 09:40:16,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.19 MB 2025-02-15 09:40:16,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-15 09:40:16,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-15 09:40:16,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:40:16,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26940.53 MB 2025-02-15 09:40:16,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:40:16,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:40:16,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.38 seconds 2025-02-15 09:40:16,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.42 MB 2025-02-15 09:40:16,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26943.18 MB 2025-02-15 09:40:16,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11723.76 MB 2025-02-15 09:40:16,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54328.82 MB 2025-02-15 09:40:16,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-15 09:40:16,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22810.72 MB 2025-02-15 09:40:16,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26943.18 MB 2025-02-15 09:40:16,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:40:16,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:40:16,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:40:16,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26943.18 MB 2025-02-15 09:40:16,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20223.81 MB 2025-02-15 09:40:16,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6719.37 MB 2025-02-15 09:40:16,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-15 09:40:16,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-15 09:40:16,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:40:16,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29454.85 MB 2025-02-15 09:40:16,879 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:40:16,879 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:40:16,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:40:16,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:40:16,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:40:16,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:16,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20223.81 MB 2025-02-15 09:40:16,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28662.83 MB 2025-02-15 09:40:16,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:40:16,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-15 09:40:16,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39908.80 MB 2025-02-15 09:40:16,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:40:16,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28662.83 MB 2025-02-15 09:40:17,050 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:40:17,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:17,052 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:40:17,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:17,053 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:40:17,057 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:40:17,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:17,058 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:40:17,059 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:40:33,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:33,017 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:40:33,022 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:40:33,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:33,026 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1506, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:40:33,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:33,027 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1506, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:40:56,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:40:56,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:40:56,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.43 seconds 2025-02-15 09:40:56,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:56,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.76 MB 2025-02-15 09:40:56,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28792.40 MB 2025-02-15 09:40:56,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5329.65 MB 2025-02-15 09:40:56,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52493.81 MB 2025-02-15 09:40:56,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-15 09:40:56,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15510.54 MB 2025-02-15 09:40:56,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37690.47 MB 2025-02-15 09:40:56,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:40:56,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:40:56,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:40:56,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:56,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28792.40 MB 2025-02-15 09:40:56,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23607.07 MB 2025-02-15 09:40:56,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5185.33 MB 2025-02-15 09:40:56,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36983.28 MB 2025-02-15 09:40:56,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 09:40:56,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4026.53 MB 2025-02-15 09:40:56,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38397.61 MB 2025-02-15 09:40:58,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:40:58,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:40:58,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 09:40:58,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23607.07 MB 2025-02-15 09:40:58,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24137.92 MB 2025-02-15 09:40:58,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:40:58,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 09:40:58,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31652.32 MB 2025-02-15 09:40:58,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9357.49 MB 2025-02-15 09:40:58,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28116.46 MB 2025-02-15 09:40:58,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:40:58,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:40:58,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:40:58,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24137.92 MB 2025-02-15 09:40:58,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26027.45 MB 2025-02-15 09:40:58,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:40:58,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31652.32 MB 2025-02-15 09:40:58,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31652.32 MB 2025-02-15 09:40:58,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:40:58,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27444.88 MB 2025-02-15 09:40:58,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:40:58,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:40:58,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:40:58,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26027.45 MB 2025-02-15 09:40:58,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28269.31 MB 2025-02-15 09:40:58,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:40:58,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31652.32 MB 2025-02-15 09:40:58,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-15 09:40:58,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 09:40:58,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33813.59 MB 2025-02-15 09:40:58,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:40:58,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:40:58,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:40:58,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24137.92 MB 2025-02-15 09:40:58,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28269.31 MB 2025-02-15 09:40:58,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:40:58,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31652.32 MB 2025-02-15 09:40:58,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-15 09:40:58,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 09:40:58,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33813.59 MB 2025-02-15 09:40:58,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:40:58,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:40:58,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:40:58,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29802.85 MB 2025-02-15 09:40:58,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30569.85 MB 2025-02-15 09:40:58,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:40:58,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35899.05 MB 2025-02-15 09:40:58,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36314.28 MB 2025-02-15 09:40:58,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:40:58,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31277.64 MB 2025-02-15 09:40:58,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:40:58,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:40:58,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:40:58,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30982.74 MB 2025-02-15 09:40:58,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31214.51 MB 2025-02-15 09:40:58,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.78 MB 2025-02-15 09:40:58,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36314.28 MB 2025-02-15 09:40:58,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36314.28 MB 2025-02-15 09:40:58,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:40:58,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31421.34 MB 2025-02-15 09:40:58,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:40:58,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:40:58,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.89 seconds 2025-02-15 09:40:58,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:58,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18215.73 MB 2025-02-15 09:40:58,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31415.59 MB 2025-02-15 09:40:58,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13199.86 MB 2025-02-15 09:40:58,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52493.81 MB 2025-02-15 09:40:58,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36314.28 MB 2025-02-15 09:40:58,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16179.53 MB 2025-02-15 09:40:58,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31421.34 MB 2025-02-15 09:40:59,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:40:59,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:40:59,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:40:59,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:59,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31415.59 MB 2025-02-15 09:40:59,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23220.12 MB 2025-02-15 09:40:59,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8195.47 MB 2025-02-15 09:40:59,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36314.28 MB 2025-02-15 09:40:59,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36314.28 MB 2025-02-15 09:40:59,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:40:59,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33927.25 MB 2025-02-15 09:40:59,203 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:40:59,203 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:40:59,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:40:59,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:40:59,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:40:59,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:40:59,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23220.12 MB 2025-02-15 09:40:59,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31659.14 MB 2025-02-15 09:40:59,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:40:59,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36314.28 MB 2025-02-15 09:40:59,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40510.69 MB 2025-02-15 09:40:59,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 09:40:59,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31659.14 MB 2025-02-15 09:40:59,373 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:40:59,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:59,375 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:40:59,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:59,375 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:40:59,380 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:40:59,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:40:59,381 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:40:59,382 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:41:40,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:40,701 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:41:40,706 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:41:40,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:40,710 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:41:40,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:40,711 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:41:46,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:41:46,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:41:46,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.85 seconds 2025-02-15 09:41:46,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:46,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-15 09:41:46,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-15 09:41:46,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-15 09:41:46,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53095.69 MB 2025-02-15 09:41:46,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19310.58 MB 2025-02-15 09:41:46,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33785.12 MB 2025-02-15 09:41:46,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-15 09:41:46,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:41:46,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:41:46,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 09:41:46,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:46,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-15 09:41:46,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17601.87 MB 2025-02-15 09:41:46,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 650.97 MB 2025-02-15 09:41:46,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19310.58 MB 2025-02-15 09:41:46,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23966.25 MB 2025-02-15 09:41:46,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4655.68 MB 2025-02-15 09:41:46,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22277.20 MB 2025-02-15 09:41:48,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:41:48,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:41:48,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.81 seconds 2025-02-15 09:41:48,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17601.87 MB 2025-02-15 09:41:48,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18104.84 MB 2025-02-15 09:41:48,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 502.97 MB 2025-02-15 09:41:48,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23966.25 MB 2025-02-15 09:41:48,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21302.87 MB 2025-02-15 09:41:48,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2663.38 MB 2025-02-15 09:41:48,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22027.36 MB 2025-02-15 09:41:48,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:41:48,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:41:48,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:41:48,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18104.84 MB 2025-02-15 09:41:48,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19895.28 MB 2025-02-15 09:41:48,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.44 MB 2025-02-15 09:41:48,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21302.87 MB 2025-02-15 09:41:48,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23989.32 MB 2025-02-15 09:41:48,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2686.45 MB 2025-02-15 09:41:48,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21238.30 MB 2025-02-15 09:41:48,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:41:48,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:41:48,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 09:41:48,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19895.28 MB 2025-02-15 09:41:48,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.45 MB 2025-02-15 09:41:48,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2124.16 MB 2025-02-15 09:41:48,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23989.32 MB 2025-02-15 09:41:48,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29364.32 MB 2025-02-15 09:41:48,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5375.00 MB 2025-02-15 09:41:48,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27272.65 MB 2025-02-15 09:41:48,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:41:48,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:41:48,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:41:48,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18104.84 MB 2025-02-15 09:41:48,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.45 MB 2025-02-15 09:41:48,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3914.61 MB 2025-02-15 09:41:48,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21302.87 MB 2025-02-15 09:41:48,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29364.32 MB 2025-02-15 09:41:48,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8061.45 MB 2025-02-15 09:41:48,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27272.65 MB 2025-02-15 09:41:48,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:41:48,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:41:48,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 09:41:48,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23472.48 MB 2025-02-15 09:41:48,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24199.21 MB 2025-02-15 09:41:48,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 726.73 MB 2025-02-15 09:41:48,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29364.32 MB 2025-02-15 09:41:48,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29756.49 MB 2025-02-15 09:41:48,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 392.17 MB 2025-02-15 09:41:48,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24869.84 MB 2025-02-15 09:41:48,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:41:48,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:41:48,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:41:48,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.43 MB 2025-02-15 09:41:48,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24801.63 MB 2025-02-15 09:41:48,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.21 MB 2025-02-15 09:41:48,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29756.49 MB 2025-02-15 09:41:48,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29758.59 MB 2025-02-15 09:41:48,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 09:41:48,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24974.14 MB 2025-02-15 09:41:48,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:41:48,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:41:48,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.09 seconds 2025-02-15 09:41:48,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:48,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-15 09:41:48,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25002.71 MB 2025-02-15 09:41:48,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10713.53 MB 2025-02-15 09:41:48,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53095.69 MB 2025-02-15 09:41:48,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29758.59 MB 2025-02-15 09:41:48,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23337.11 MB 2025-02-15 09:41:48,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25002.71 MB 2025-02-15 09:41:49,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:41:49,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:41:49,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:41:49,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:49,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25002.71 MB 2025-02-15 09:41:49,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19194.06 MB 2025-02-15 09:41:49,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5808.64 MB 2025-02-15 09:41:49,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29758.59 MB 2025-02-15 09:41:49,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29758.59 MB 2025-02-15 09:41:49,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:41:49,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27815.77 MB 2025-02-15 09:41:49,091 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:41:49,092 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:41:49,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:41:49,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:41:49,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:41:49,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:41:49,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19194.06 MB 2025-02-15 09:41:49,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27633.09 MB 2025-02-15 09:41:49,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:41:49,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29758.59 MB 2025-02-15 09:41:49,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40248.54 MB 2025-02-15 09:41:49,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 09:41:49,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27633.09 MB 2025-02-15 09:41:49,261 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:41:49,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:49,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:41:49,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:49,263 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:41:49,268 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:41:49,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:41:49,269 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:41:49,269 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:43:08,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:08,972 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:43:08,977 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:43:08,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:08,981 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 761, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:43:08,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:08,982 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 761, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:43:20,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:43:20,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:43:20,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.74 seconds 2025-02-15 09:43:20,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:20,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18271.48 MB 2025-02-15 09:43:20,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20964.61 MB 2025-02-15 09:43:20,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2693.14 MB 2025-02-15 09:43:20,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 09:43:20,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24115.15 MB 2025-02-15 09:43:20,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28718.40 MB 2025-02-15 09:43:20,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.28 MB 2025-02-15 09:43:20,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:43:20,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:43:20,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:43:20,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:20,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20964.61 MB 2025-02-15 09:43:20,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19734.05 MB 2025-02-15 09:43:20,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1230.56 MB 2025-02-15 09:43:20,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24115.15 MB 2025-02-15 09:43:20,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29882.32 MB 2025-02-15 09:43:20,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5767.17 MB 2025-02-15 09:43:20,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29537.57 MB 2025-02-15 09:43:22,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:43:22,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:43:22,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 09:43:22,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:22,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19734.05 MB 2025-02-15 09:43:22,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20264.89 MB 2025-02-15 09:43:22,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:43:22,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29882.32 MB 2025-02-15 09:43:22,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25530.73 MB 2025-02-15 09:43:22,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4351.59 MB 2025-02-15 09:43:22,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24243.44 MB 2025-02-15 09:43:22,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:43:22,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:43:22,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:43:22,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:22,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.89 MB 2025-02-15 09:43:22,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22154.43 MB 2025-02-15 09:43:22,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:43:22,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25530.73 MB 2025-02-15 09:43:22,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25530.73 MB 2025-02-15 09:43:22,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:43:22,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23571.86 MB 2025-02-15 09:43:22,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:43:22,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:43:22,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:43:22,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:22,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22154.43 MB 2025-02-15 09:43:22,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24396.28 MB 2025-02-15 09:43:22,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:43:22,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25530.73 MB 2025-02-15 09:43:22,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32136.76 MB 2025-02-15 09:43:22,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:43:22,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.57 MB 2025-02-15 09:43:22,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:43:22,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:43:22,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:43:22,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:22,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.89 MB 2025-02-15 09:43:22,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24396.28 MB 2025-02-15 09:43:22,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:43:22,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25530.73 MB 2025-02-15 09:43:22,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32136.76 MB 2025-02-15 09:43:22,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:43:22,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.57 MB 2025-02-15 09:43:23,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:43:23,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:43:23,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:43:23,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:23,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.83 MB 2025-02-15 09:43:23,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26696.83 MB 2025-02-15 09:43:23,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:43:23,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32136.76 MB 2025-02-15 09:43:23,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32551.99 MB 2025-02-15 09:43:23,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:43:23,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27404.62 MB 2025-02-15 09:43:23,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:43:23,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:43:23,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:43:23,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:23,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27109.72 MB 2025-02-15 09:43:23,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.00 MB 2025-02-15 09:43:23,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-15 09:43:23,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32551.99 MB 2025-02-15 09:43:23,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32551.99 MB 2025-02-15 09:43:23,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:43:23,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27513.58 MB 2025-02-15 09:43:23,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:43:23,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:43:23,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.12 seconds 2025-02-15 09:43:23,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:23,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15620.09 MB 2025-02-15 09:43:23,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27538.85 MB 2025-02-15 09:43:23,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11918.76 MB 2025-02-15 09:43:23,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 09:43:23,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32551.99 MB 2025-02-15 09:43:23,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20281.56 MB 2025-02-15 09:43:23,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27538.85 MB 2025-02-15 09:43:23,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:43:23,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:43:23,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:43:23,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:23,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27538.85 MB 2025-02-15 09:43:23,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20613.21 MB 2025-02-15 09:43:23,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6925.64 MB 2025-02-15 09:43:23,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32551.99 MB 2025-02-15 09:43:23,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32551.99 MB 2025-02-15 09:43:23,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:43:23,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30041.00 MB 2025-02-15 09:43:23,396 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 09:43:23,396 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:43:23,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:43:23,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:43:23,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:43:23,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:43:23,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20613.21 MB 2025-02-15 09:43:23,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29020.95 MB 2025-02-15 09:43:23,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 09:43:23,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32551.99 MB 2025-02-15 09:43:23,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36731.62 MB 2025-02-15 09:43:23,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 09:43:23,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29020.95 MB 2025-02-15 09:43:23,564 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 09:43:23,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:23,566 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:43:23,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:23,566 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:43:23,571 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:43:23,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:43:23,572 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:43:23,572 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:44:10,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:10,334 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:44:10,340 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:44:10,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:10,344 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1894, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:44:10,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:10,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1894, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:44:39,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:44:39,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:44:39,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.22 seconds 2025-02-15 09:44:39,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:39,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26166.40 MB 2025-02-15 09:44:39,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32869.16 MB 2025-02-15 09:44:39,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6702.76 MB 2025-02-15 09:44:39,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45090.87 MB 2025-02-15 09:44:39,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39919.29 MB 2025-02-15 09:44:39,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5171.58 MB 2025-02-15 09:44:39,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41753.07 MB 2025-02-15 09:44:39,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:44:39,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:44:39,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 09:44:39,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:39,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32869.16 MB 2025-02-15 09:44:39,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25624.17 MB 2025-02-15 09:44:39,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7245.00 MB 2025-02-15 09:44:39,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39919.29 MB 2025-02-15 09:44:39,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53550.78 MB 2025-02-15 09:44:39,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13631.49 MB 2025-02-15 09:44:39,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51238.37 MB 2025-02-15 09:44:41,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:44:41,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:44:41,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:44:41,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:41,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.17 MB 2025-02-15 09:44:41,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26155.01 MB 2025-02-15 09:44:41,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:44:41,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53550.78 MB 2025-02-15 09:44:41,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-15 09:44:41,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18920.51 MB 2025-02-15 09:44:41,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30133.56 MB 2025-02-15 09:44:41,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:44:41,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:44:41,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:44:41,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:41,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.01 MB 2025-02-15 09:44:41,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28044.54 MB 2025-02-15 09:44:41,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:44:41,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-15 09:44:41,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-15 09:44:41,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:44:41,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29461.97 MB 2025-02-15 09:44:41,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:44:41,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:44:41,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:44:41,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:41,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28044.54 MB 2025-02-15 09:44:41,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30286.40 MB 2025-02-15 09:44:41,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:44:41,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-15 09:44:41,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 09:44:41,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 09:44:41,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35830.68 MB 2025-02-15 09:44:41,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:44:41,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:44:41,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:44:41,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:41,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.01 MB 2025-02-15 09:44:41,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30286.40 MB 2025-02-15 09:44:41,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:44:41,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-15 09:44:41,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 09:44:41,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 09:44:41,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35830.68 MB 2025-02-15 09:44:42,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:44:42,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:44:42,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:44:42,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:42,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31819.94 MB 2025-02-15 09:44:42,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32586.94 MB 2025-02-15 09:44:42,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:44:42,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-15 09:44:42,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 09:44:42,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:44:42,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33294.73 MB 2025-02-15 09:44:42,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:44:42,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:44:42,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:44:42,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:42,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32999.83 MB 2025-02-15 09:44:42,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33228.28 MB 2025-02-15 09:44:42,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 09:44:42,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 09:44:42,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 09:44:42,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:44:42,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33435.76 MB 2025-02-15 09:44:42,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:44:42,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:44:42,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.72 seconds 2025-02-15 09:44:42,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:42,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19567.55 MB 2025-02-15 09:44:42,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33428.71 MB 2025-02-15 09:44:42,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13861.16 MB 2025-02-15 09:44:42,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45090.87 MB 2025-02-15 09:44:42,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 09:44:42,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6272.58 MB 2025-02-15 09:44:42,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33435.76 MB 2025-02-15 09:44:42,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:44:42,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:44:42,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:44:42,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:42,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33428.71 MB 2025-02-15 09:44:42,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24562.46 MB 2025-02-15 09:44:42,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8866.25 MB 2025-02-15 09:44:42,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 09:44:42,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-15 09:44:42,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:44:42,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35932.81 MB 2025-02-15 09:44:42,357 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 09:44:42,357 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:44:42,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:44:42,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:44:42,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:44:42,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:44:42,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24562.46 MB 2025-02-15 09:44:42,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32974.31 MB 2025-02-15 09:44:42,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8411.85 MB 2025-02-15 09:44:42,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-15 09:44:42,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43000.00 MB 2025-02-15 09:44:42,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-15 09:44:42,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32974.31 MB 2025-02-15 09:44:42,522 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 09:44:42,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:42,523 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:44:42,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:42,524 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:44:42,529 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:44:42,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:44:42,530 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:44:42,530 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:45:04,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:04,211 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:45:04,216 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:45:04,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:04,220 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1087, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:45:04,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:04,221 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1087, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:45:21,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:45:21,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:45:21,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.94 seconds 2025-02-15 09:45:21,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:21,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20543.10 MB 2025-02-15 09:45:21,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24389.93 MB 2025-02-15 09:45:21,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3846.83 MB 2025-02-15 09:45:21,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51363.45 MB 2025-02-15 09:45:21,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28689.04 MB 2025-02-15 09:45:21,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22674.41 MB 2025-02-15 09:45:21,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33268.09 MB 2025-02-15 09:45:21,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:45:21,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:45:21,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:45:21,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:21,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24389.93 MB 2025-02-15 09:45:21,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21429.87 MB 2025-02-15 09:45:21,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2960.06 MB 2025-02-15 09:45:21,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28689.04 MB 2025-02-15 09:45:21,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37381.73 MB 2025-02-15 09:45:21,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8692.70 MB 2025-02-15 09:45:21,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35182.88 MB 2025-02-15 09:45:23,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:45:23,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:45:23,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:45:23,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21429.87 MB 2025-02-15 09:45:23,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21960.72 MB 2025-02-15 09:45:23,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:45:23,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37381.73 MB 2025-02-15 09:45:23,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26965.18 MB 2025-02-15 09:45:23,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10416.55 MB 2025-02-15 09:45:23,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25939.26 MB 2025-02-15 09:45:23,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:45:23,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:45:23,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:45:23,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21960.72 MB 2025-02-15 09:45:23,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23850.25 MB 2025-02-15 09:45:23,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:45:23,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26965.18 MB 2025-02-15 09:45:23,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27908.90 MB 2025-02-15 09:45:23,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:45:23,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25267.68 MB 2025-02-15 09:45:23,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:45:23,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:45:23,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:45:23,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23850.25 MB 2025-02-15 09:45:23,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26092.11 MB 2025-02-15 09:45:23,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:45:23,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27908.90 MB 2025-02-15 09:45:23,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-15 09:45:23,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:45:23,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31636.39 MB 2025-02-15 09:45:23,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:45:23,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:45:23,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:45:23,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21960.72 MB 2025-02-15 09:45:23,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26092.11 MB 2025-02-15 09:45:23,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:45:23,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26965.18 MB 2025-02-15 09:45:23,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-15 09:45:23,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:45:23,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31636.39 MB 2025-02-15 09:45:23,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:45:23,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:45:23,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:45:23,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27625.65 MB 2025-02-15 09:45:23,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28392.65 MB 2025-02-15 09:45:23,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:45:23,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33571.21 MB 2025-02-15 09:45:23,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-15 09:45:23,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:45:23,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29100.44 MB 2025-02-15 09:45:23,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:45:23,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:45:23,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:45:23,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28805.54 MB 2025-02-15 09:45:23,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29034.44 MB 2025-02-15 09:45:23,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-15 09:45:23,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33984.35 MB 2025-02-15 09:45:23,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-15 09:45:23,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:45:23,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29268.35 MB 2025-02-15 09:45:23,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:45:23,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:45:23,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.37 seconds 2025-02-15 09:45:23,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16755.90 MB 2025-02-15 09:45:23,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29235.39 MB 2025-02-15 09:45:23,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12479.49 MB 2025-02-15 09:45:23,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51363.45 MB 2025-02-15 09:45:23,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-15 09:45:23,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17379.10 MB 2025-02-15 09:45:23,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29268.35 MB 2025-02-15 09:45:23,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:45:23,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:45:23,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:45:23,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29235.39 MB 2025-02-15 09:45:23,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21758.39 MB 2025-02-15 09:45:23,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7477.00 MB 2025-02-15 09:45:23,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33984.35 MB 2025-02-15 09:45:23,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-15 09:45:23,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:45:23,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31745.52 MB 2025-02-15 09:45:23,876 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 09:45:23,876 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:45:23,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:45:23,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:45:23,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:45:23,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:45:23,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21758.39 MB 2025-02-15 09:45:23,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30193.00 MB 2025-02-15 09:45:23,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 09:45:23,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33984.35 MB 2025-02-15 09:45:23,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42368.76 MB 2025-02-15 09:45:23,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 09:45:23,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.00 MB 2025-02-15 09:45:24,041 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 09:45:24,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:24,043 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:45:24,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:24,044 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:45:24,048 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:45:24,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:45:24,049 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:45:24,049 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 09:46:29,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:29,597 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:46:29,602 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:46:29,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:29,606 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 469, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:46:29,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:29,607 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 469, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:46:36,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:46:36,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:46:36,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.27 seconds 2025-02-15 09:46:36,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:36,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16236.77 MB 2025-02-15 09:46:36,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17896.54 MB 2025-02-15 09:46:36,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1659.76 MB 2025-02-15 09:46:36,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50753.18 MB 2025-02-15 09:46:36,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20256.39 MB 2025-02-15 09:46:36,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30496.78 MB 2025-02-15 09:46:36,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26840.61 MB 2025-02-15 09:46:36,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:46:36,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:46:36,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:46:36,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:36,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17896.54 MB 2025-02-15 09:46:36,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18217.08 MB 2025-02-15 09:46:36,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.55 MB 2025-02-15 09:46:36,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20256.39 MB 2025-02-15 09:46:36,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25369.25 MB 2025-02-15 09:46:36,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5112.86 MB 2025-02-15 09:46:36,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25108.01 MB 2025-02-15 09:46:38,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:46:38,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:46:38,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 09:46:38,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:38,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18217.08 MB 2025-02-15 09:46:38,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18747.93 MB 2025-02-15 09:46:38,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:46:38,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25369.25 MB 2025-02-15 09:46:38,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21437.09 MB 2025-02-15 09:46:38,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3932.16 MB 2025-02-15 09:46:38,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22728.55 MB 2025-02-15 09:46:38,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:46:38,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:46:38,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:46:38,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:38,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-15 09:46:38,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20637.46 MB 2025-02-15 09:46:38,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:46:38,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21437.09 MB 2025-02-15 09:46:38,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24268.24 MB 2025-02-15 09:46:38,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:46:38,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22054.89 MB 2025-02-15 09:46:39,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:46:39,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:46:39,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:46:39,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20637.46 MB 2025-02-15 09:46:39,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-15 09:46:39,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:46:39,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24268.24 MB 2025-02-15 09:46:39,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30402.41 MB 2025-02-15 09:46:39,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:46:39,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-15 09:46:39,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:46:39,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:46:39,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:46:39,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-15 09:46:39,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-15 09:46:39,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:46:39,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21437.09 MB 2025-02-15 09:46:39,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30402.41 MB 2025-02-15 09:46:39,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:46:39,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-15 09:46:39,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:46:39,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:46:39,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:46:39,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24412.86 MB 2025-02-15 09:46:39,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25179.86 MB 2025-02-15 09:46:39,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:46:39,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30402.41 MB 2025-02-15 09:46:39,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:46:39,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:46:39,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25887.65 MB 2025-02-15 09:46:39,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:46:39,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:46:39,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:46:39,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25592.75 MB 2025-02-15 09:46:39,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.31 MB 2025-02-15 09:46:39,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.56 MB 2025-02-15 09:46:39,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:46:39,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:46:39,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:46:39,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25985.75 MB 2025-02-15 09:46:39,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:46:39,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:46:39,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.66 seconds 2025-02-15 09:46:39,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14602.74 MB 2025-02-15 09:46:39,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26021.38 MB 2025-02-15 09:46:39,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11418.64 MB 2025-02-15 09:46:39,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50753.18 MB 2025-02-15 09:46:39,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:46:39,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19937.62 MB 2025-02-15 09:46:39,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26021.38 MB 2025-02-15 09:46:39,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:46:39,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:46:39,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:46:39,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26021.38 MB 2025-02-15 09:46:39,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19607.13 MB 2025-02-15 09:46:39,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6414.25 MB 2025-02-15 09:46:39,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:46:39,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 09:46:39,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:46:39,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28533.05 MB 2025-02-15 09:46:39,559 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:46:39,559 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:46:39,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:46:39,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:46:39,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:46:39,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:46:39,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19607.13 MB 2025-02-15 09:46:39,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28046.15 MB 2025-02-15 09:46:39,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:46:39,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 09:46:39,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 09:46:39,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 09:46:39,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28046.15 MB 2025-02-15 09:46:39,727 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:46:39,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:39,729 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:46:39,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:39,730 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:46:39,735 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:46:39,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:46:39,736 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:46:39,736 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:47:46,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:47:46,182 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:47:46,188 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:47:46,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:47:46,194 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1502, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:47:46,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:47:46,196 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1502, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:48:09,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:48:09,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:48:09,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.23 seconds 2025-02-15 09:48:09,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:09,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23434.88 MB 2025-02-15 09:48:09,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28751.16 MB 2025-02-15 09:48:09,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5316.28 MB 2025-02-15 09:48:09,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 09:48:09,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38575.01 MB 2025-02-15 09:48:09,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15315.50 MB 2025-02-15 09:48:09,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37662.59 MB 2025-02-15 09:48:09,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:48:09,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:48:09,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:48:09,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:09,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28751.16 MB 2025-02-15 09:48:09,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23586.28 MB 2025-02-15 09:48:09,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5164.88 MB 2025-02-15 09:48:09,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38575.01 MB 2025-02-15 09:48:09,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47903.15 MB 2025-02-15 09:48:09,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9328.13 MB 2025-02-15 09:48:09,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42781.62 MB 2025-02-15 09:48:11,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:48:11,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:48:11,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:48:11,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23586.28 MB 2025-02-15 09:48:11,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24117.12 MB 2025-02-15 09:48:11,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:48:11,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47903.15 MB 2025-02-15 09:48:11,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 09:48:11,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18838.72 MB 2025-02-15 09:48:11,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28095.67 MB 2025-02-15 09:48:11,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:48:11,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:48:11,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:48:11,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-15 09:48:11,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26006.66 MB 2025-02-15 09:48:11,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:48:11,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 09:48:11,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29064.43 MB 2025-02-15 09:48:11,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:11,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.08 MB 2025-02-15 09:48:11,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:48:11,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:48:11,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:48:11,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26006.66 MB 2025-02-15 09:48:11,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-15 09:48:11,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:48:11,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 09:48:11,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36379.30 MB 2025-02-15 09:48:11,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 09:48:11,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-15 09:48:11,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:48:11,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:48:11,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:48:11,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-15 09:48:11,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-15 09:48:11,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:48:11,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29064.43 MB 2025-02-15 09:48:11,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36379.30 MB 2025-02-15 09:48:11,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 09:48:11,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-15 09:48:11,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:48:11,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:48:11,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:48:11,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29782.05 MB 2025-02-15 09:48:11,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30549.06 MB 2025-02-15 09:48:11,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:48:11,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36379.30 MB 2025-02-15 09:48:11,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 09:48:11,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:48:11,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31256.84 MB 2025-02-15 09:48:11,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:48:11,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:48:11,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:48:11,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30961.95 MB 2025-02-15 09:48:11,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31192.25 MB 2025-02-15 09:48:11,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.30 MB 2025-02-15 09:48:11,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 09:48:11,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 09:48:11,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:11,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31387.99 MB 2025-02-15 09:48:11,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:48:11,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:48:11,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.67 seconds 2025-02-15 09:48:11,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:11,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18201.79 MB 2025-02-15 09:48:11,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31393.32 MB 2025-02-15 09:48:11,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13191.52 MB 2025-02-15 09:48:11,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 09:48:11,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 09:48:11,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17095.98 MB 2025-02-15 09:48:11,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31393.32 MB 2025-02-15 09:48:12,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:48:12,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:48:12,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:48:12,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:12,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31393.32 MB 2025-02-15 09:48:12,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23206.18 MB 2025-02-15 09:48:12,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8187.13 MB 2025-02-15 09:48:12,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 09:48:12,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 09:48:12,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:12,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33904.99 MB 2025-02-15 09:48:12,157 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:48:12,158 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:48:12,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:48:12,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:48:12,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 09:48:12,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:12,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23206.18 MB 2025-02-15 09:48:12,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31645.21 MB 2025-02-15 09:48:12,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:48:12,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 09:48:12,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45185.24 MB 2025-02-15 09:48:12,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:48:12,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31645.21 MB 2025-02-15 09:48:12,340 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:48:12,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:12,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:48:12,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:12,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:48:12,347 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:48:12,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:12,348 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:48:12,348 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:48:21,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:21,801 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:48:21,806 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:48:21,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:21,810 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1446, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:48:21,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:21,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1446, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:48:44,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:48:44,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:48:44,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.55 seconds 2025-02-15 09:48:44,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:44,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23044.67 MB 2025-02-15 09:48:44,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28161.98 MB 2025-02-15 09:48:44,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5117.31 MB 2025-02-15 09:48:44,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57770.25 MB 2025-02-15 09:48:44,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-15 09:48:44,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19396.56 MB 2025-02-15 09:48:44,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37045.89 MB 2025-02-15 09:48:44,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:48:44,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:48:44,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:48:44,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:44,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28161.98 MB 2025-02-15 09:48:44,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23295.15 MB 2025-02-15 09:48:44,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4866.83 MB 2025-02-15 09:48:44,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-15 09:48:44,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48129.64 MB 2025-02-15 09:48:44,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9755.95 MB 2025-02-15 09:48:44,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42833.57 MB 2025-02-15 09:48:46,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:48:46,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:48:46,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:48:46,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23295.15 MB 2025-02-15 09:48:46,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23826.00 MB 2025-02-15 09:48:46,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:48:46,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48129.64 MB 2025-02-15 09:48:46,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 09:48:46,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19069.40 MB 2025-02-15 09:48:46,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27804.54 MB 2025-02-15 09:48:46,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:48:46,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:48:46,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:48:46,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23826.00 MB 2025-02-15 09:48:46,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25715.53 MB 2025-02-15 09:48:46,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:48:46,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 09:48:46,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30003.95 MB 2025-02-15 09:48:46,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 09:48:46,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27132.96 MB 2025-02-15 09:48:46,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:48:46,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:48:46,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:48:46,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25715.53 MB 2025-02-15 09:48:46,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27957.39 MB 2025-02-15 09:48:46,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:48:46,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30003.95 MB 2025-02-15 09:48:46,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35666.26 MB 2025-02-15 09:48:46,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 09:48:46,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33501.67 MB 2025-02-15 09:48:46,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:48:46,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:48:46,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:48:46,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23826.00 MB 2025-02-15 09:48:46,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27957.39 MB 2025-02-15 09:48:46,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:48:46,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 09:48:46,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35666.26 MB 2025-02-15 09:48:46,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 09:48:46,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33501.67 MB 2025-02-15 09:48:46,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:48:46,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:48:46,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:48:46,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29490.93 MB 2025-02-15 09:48:46,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30257.93 MB 2025-02-15 09:48:46,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:48:46,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35666.26 MB 2025-02-15 09:48:46,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-15 09:48:46,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:48:46,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30965.72 MB 2025-02-15 09:48:46,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:48:46,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:48:46,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:48:46,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30670.82 MB 2025-02-15 09:48:46,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30899.09 MB 2025-02-15 09:48:46,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-15 09:48:46,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36083.60 MB 2025-02-15 09:48:46,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-15 09:48:46,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:46,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31117.02 MB 2025-02-15 09:48:46,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:48:46,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:48:46,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.98 seconds 2025-02-15 09:48:46,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:46,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18006.69 MB 2025-02-15 09:48:46,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31099.94 MB 2025-02-15 09:48:46,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13093.26 MB 2025-02-15 09:48:46,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57770.25 MB 2025-02-15 09:48:46,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-15 09:48:46,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21686.65 MB 2025-02-15 09:48:46,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31117.02 MB 2025-02-15 09:48:47,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:48:47,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:48:47,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:48:47,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:47,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31099.94 MB 2025-02-15 09:48:47,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23001.23 MB 2025-02-15 09:48:47,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8098.71 MB 2025-02-15 09:48:47,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36083.60 MB 2025-02-15 09:48:47,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-15 09:48:47,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:47,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33603.32 MB 2025-02-15 09:48:47,085 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-15 09:48:47,086 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:48:47,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:48:47,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:48:47,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:48:47,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:47,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23001.23 MB 2025-02-15 09:48:47,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31412.05 MB 2025-02-15 09:48:47,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-15 09:48:47,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36083.60 MB 2025-02-15 09:48:47,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44447.04 MB 2025-02-15 09:48:47,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 09:48:47,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31412.05 MB 2025-02-15 09:48:47,252 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-15 09:48:47,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:47,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:48:47,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:47,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:48:47,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:48:47,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:47,260 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:48:47,260 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:48:55,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:55,815 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:48:55,819 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:48:55,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:55,823 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:48:55,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:55,824 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:48:58,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:48:58,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:48:58,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.39 seconds 2025-02-15 09:48:58,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:58,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-15 09:48:58,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-15 09:48:58,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-15 09:48:58,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52810.48 MB 2025-02-15 09:48:58,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:58,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30719.08 MB 2025-02-15 09:48:58,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23499.24 MB 2025-02-15 09:48:58,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:48:58,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:48:58,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:48:58,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:58,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-15 09:48:58,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.27 MB 2025-02-15 09:48:58,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.48 MB 2025-02-15 09:48:58,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:58,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:58,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:58,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16616.57 MB 2025-02-15 09:48:58,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:48:58,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:48:58,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-15 09:48:58,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:58,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.27 MB 2025-02-15 09:48:58,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14978.03 MB 2025-02-15 09:48:58,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-15 09:48:58,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:58,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:58,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:58,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18953.92 MB 2025-02-15 09:48:58,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:48:58,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:48:58,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:48:58,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:58,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-15 09:48:58,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15667.47 MB 2025-02-15 09:48:58,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-15 09:48:58,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:58,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:58,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:58,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16184.84 MB 2025-02-15 09:48:59,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:48:59,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:48:59,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:48:59,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15667.47 MB 2025-02-15 09:48:59,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-15 09:48:59,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-15 09:48:59,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:59,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:59,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:59,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-15 09:48:59,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:48:59,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:48:59,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:48:59,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-15 09:48:59,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-15 09:48:59,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-15 09:48:59,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:59,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 09:48:59,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:59,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-15 09:48:59,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:48:59,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:48:59,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:48:59,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17045.54 MB 2025-02-15 09:48:59,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.49 MB 2025-02-15 09:48:59,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-15 09:48:59,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 09:48:59,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22238.20 MB 2025-02-15 09:48:59,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-15 09:48:59,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17594.53 MB 2025-02-15 09:48:59,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:48:59,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:48:59,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:48:59,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.20 MB 2025-02-15 09:48:59,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17681.56 MB 2025-02-15 09:48:59,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.36 MB 2025-02-15 09:48:59,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22238.20 MB 2025-02-15 09:48:59,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22242.39 MB 2025-02-15 09:48:59,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 09:48:59,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17688.30 MB 2025-02-15 09:48:59,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:48:59,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:48:59,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-15 09:48:59,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-15 09:48:59,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17882.59 MB 2025-02-15 09:48:59,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4384.30 MB 2025-02-15 09:48:59,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52810.48 MB 2025-02-15 09:48:59,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22242.39 MB 2025-02-15 09:48:59,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30568.09 MB 2025-02-15 09:48:59,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17882.59 MB 2025-02-15 09:48:59,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:48:59,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:48:59,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:48:59,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17882.59 MB 2025-02-15 09:48:59,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17303.36 MB 2025-02-15 09:48:59,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -579.22 MB 2025-02-15 09:48:59,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22242.39 MB 2025-02-15 09:48:59,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22242.39 MB 2025-02-15 09:48:59,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:48:59,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18987.45 MB 2025-02-15 09:48:59,395 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 09:48:59,395 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-15 09:48:59,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:48:59,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:48:59,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:48:59,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:48:59,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17303.36 MB 2025-02-15 09:48:59,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25740.83 MB 2025-02-15 09:48:59,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 09:48:59,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22242.39 MB 2025-02-15 09:48:59,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30631.00 MB 2025-02-15 09:48:59,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 09:48:59,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25740.83 MB 2025-02-15 09:48:59,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 09:48:59,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:59,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:48:59,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:59,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:48:59,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:48:59,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:48:59,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:48:59,569 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-15 09:50:40,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:40,951 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:50:40,956 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:50:40,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:40,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 128, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:50:40,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:40,963 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 128, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:50:42,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:50:42,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:50:42,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 09:50:42,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:42,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13860.63 MB 2025-02-15 09:50:42,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14313.62 MB 2025-02-15 09:50:42,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 452.98 MB 2025-02-15 09:50:42,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39019.61 MB 2025-02-15 09:50:42,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:42,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21126.71 MB 2025-02-15 09:50:42,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23332.00 MB 2025-02-15 09:50:42,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:50:42,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:50:42,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:50:42,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:42,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.62 MB 2025-02-15 09:50:42,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14252.16 MB 2025-02-15 09:50:42,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -61.45 MB 2025-02-15 09:50:42,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:42,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:42,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:42,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15549.73 MB 2025-02-15 09:50:43,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:50:43,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:50:43,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.43 seconds 2025-02-15 09:50:43,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14252.16 MB 2025-02-15 09:50:43,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14368.95 MB 2025-02-15 09:50:43,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 116.79 MB 2025-02-15 09:50:43,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:43,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:43,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18337.92 MB 2025-02-15 09:50:43,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:50:43,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:50:43,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:50:43,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14368.88 MB 2025-02-15 09:50:43,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.48 MB 2025-02-15 09:50:43,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 415.60 MB 2025-02-15 09:50:43,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:43,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:43,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15096.32 MB 2025-02-15 09:50:43,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:50:43,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:50:43,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:50:43,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.48 MB 2025-02-15 09:50:43,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15289.27 MB 2025-02-15 09:50:43,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 504.79 MB 2025-02-15 09:50:43,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:43,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:43,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16497.43 MB 2025-02-15 09:50:43,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:50:43,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:50:43,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:50:43,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14368.88 MB 2025-02-15 09:50:43,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15289.27 MB 2025-02-15 09:50:43,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.39 MB 2025-02-15 09:50:43,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:43,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-15 09:50:43,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16497.43 MB 2025-02-15 09:50:43,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:50:43,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:50:43,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:50:43,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.60 MB 2025-02-15 09:50:43,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15988.59 MB 2025-02-15 09:50:43,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.99 MB 2025-02-15 09:50:43,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-15 09:50:43,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18029.22 MB 2025-02-15 09:50:43,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-15 09:50:43,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16144.31 MB 2025-02-15 09:50:43,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:50:43,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:50:43,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:50:43,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16122.69 MB 2025-02-15 09:50:43,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16333.17 MB 2025-02-15 09:50:43,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.48 MB 2025-02-15 09:50:43,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18029.22 MB 2025-02-15 09:50:43,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18029.22 MB 2025-02-15 09:50:43,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16333.17 MB 2025-02-15 09:50:43,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:50:43,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:50:43,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.56 seconds 2025-02-15 09:50:43,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13414.67 MB 2025-02-15 09:50:43,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16520.79 MB 2025-02-15 09:50:43,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3106.12 MB 2025-02-15 09:50:43,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39019.61 MB 2025-02-15 09:50:43,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18029.22 MB 2025-02-15 09:50:43,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20990.39 MB 2025-02-15 09:50:43,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16520.79 MB 2025-02-15 09:50:43,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:50:43,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:50:43,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 09:50:43,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13925.86 MB 2025-02-15 09:50:43,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16738.25 MB 2025-02-15 09:50:43,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2812.39 MB 2025-02-15 09:50:43,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18029.22 MB 2025-02-15 09:50:43,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18029.22 MB 2025-02-15 09:50:43,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:50:43,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17019.45 MB 2025-02-15 09:50:43,797 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7615, cut from 7617 2025-02-15 09:50:43,798 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:50:43,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:50:43,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:50:43,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:50:43,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:50:43,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16738.25 MB 2025-02-15 09:50:43,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24612.33 MB 2025-02-15 09:50:43,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.08 MB 2025-02-15 09:50:43,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18029.22 MB 2025-02-15 09:50:43,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27818.72 MB 2025-02-15 09:50:43,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9789.51 MB 2025-02-15 09:50:43,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24612.33 MB 2025-02-15 09:50:43,956 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7407] 2025-02-15 09:50:43,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:43,957 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:50:43,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:43,958 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:50:43,963 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:50:43,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:50:43,964 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:50:43,964 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:51:34,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:51:34,637 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:51:34,645 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:51:34,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:51:34,652 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2364, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:51:34,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:51:34,654 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2364, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:52:11,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:52:11,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:52:11,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.64 seconds 2025-02-15 09:52:11,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:11,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29443.93 MB 2025-02-15 09:52:11,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37810.00 MB 2025-02-15 09:52:11,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8366.06 MB 2025-02-15 09:52:11,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52124.71 MB 2025-02-15 09:52:11,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41320.19 MB 2025-02-15 09:52:11,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10804.53 MB 2025-02-15 09:52:11,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46616.05 MB 2025-02-15 09:52:11,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:52:11,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:52:11,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:52:11,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:11,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37810.00 MB 2025-02-15 09:52:11,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28070.31 MB 2025-02-15 09:52:11,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9739.69 MB 2025-02-15 09:52:11,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41320.19 MB 2025-02-15 09:52:11,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59152.27 MB 2025-02-15 09:52:11,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17832.08 MB 2025-02-15 09:52:11,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61429.95 MB 2025-02-15 09:52:13,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:52:13,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:52:13,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:52:13,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28070.31 MB 2025-02-15 09:52:13,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28601.15 MB 2025-02-15 09:52:13,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:52:13,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59152.27 MB 2025-02-15 09:52:13,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30893.15 MB 2025-02-15 09:52:13,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28259.12 MB 2025-02-15 09:52:13,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32579.70 MB 2025-02-15 09:52:13,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:52:13,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:52:13,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:52:13,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28601.15 MB 2025-02-15 09:52:13,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30490.68 MB 2025-02-15 09:52:13,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:52:13,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-15 09:52:13,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33724.30 MB 2025-02-15 09:52:13,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:52:13,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31908.11 MB 2025-02-15 09:52:13,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:52:13,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:52:13,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:52:13,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30490.68 MB 2025-02-15 09:52:13,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32732.54 MB 2025-02-15 09:52:13,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:52:13,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33724.30 MB 2025-02-15 09:52:13,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39858.47 MB 2025-02-15 09:52:13,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:52:13,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38276.82 MB 2025-02-15 09:52:13,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:52:13,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:52:13,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:52:13,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28601.15 MB 2025-02-15 09:52:13,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32732.54 MB 2025-02-15 09:52:13,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:52:13,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-15 09:52:13,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39858.47 MB 2025-02-15 09:52:13,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 09:52:13,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38276.82 MB 2025-02-15 09:52:13,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:52:13,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:52:13,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:52:13,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34266.08 MB 2025-02-15 09:52:13,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35033.08 MB 2025-02-15 09:52:13,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:52:13,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39858.47 MB 2025-02-15 09:52:13,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40275.80 MB 2025-02-15 09:52:13,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:52:13,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35740.87 MB 2025-02-15 09:52:13,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:52:13,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:52:13,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:52:13,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35445.97 MB 2025-02-15 09:52:13,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35674.83 MB 2025-02-15 09:52:13,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-15 09:52:13,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40275.80 MB 2025-02-15 09:52:13,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40275.80 MB 2025-02-15 09:52:13,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:52:13,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35894.62 MB 2025-02-15 09:52:13,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:52:13,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:52:13,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.24 seconds 2025-02-15 09:52:13,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:13,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21206.32 MB 2025-02-15 09:52:13,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35875.61 MB 2025-02-15 09:52:13,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14669.29 MB 2025-02-15 09:52:13,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43887.10 MB 2025-02-15 09:52:13,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40275.80 MB 2025-02-15 09:52:13,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3611.30 MB 2025-02-15 09:52:13,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35894.62 MB 2025-02-15 09:52:14,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:52:14,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:52:14,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:52:14,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:14,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35875.61 MB 2025-02-15 09:52:14,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26206.21 MB 2025-02-15 09:52:14,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9669.40 MB 2025-02-15 09:52:14,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40275.80 MB 2025-02-15 09:52:14,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40275.80 MB 2025-02-15 09:52:14,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:52:14,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38383.59 MB 2025-02-15 09:52:14,187 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 09:52:14,187 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:52:14,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:52:14,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:52:14,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:52:14,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:52:14,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26206.21 MB 2025-02-15 09:52:14,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34632.71 MB 2025-02-15 09:52:14,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 09:52:14,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40275.80 MB 2025-02-15 09:52:14,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48653.93 MB 2025-02-15 09:52:14,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-15 09:52:14,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34632.71 MB 2025-02-15 09:52:14,351 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 09:52:14,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:52:14,353 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:52:14,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:52:14,354 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:52:14,358 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:52:14,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:52:14,359 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:52:14,359 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:53:40,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:53:40,727 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:53:40,735 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:53:40,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:53:40,742 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:53:40,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:53:40,744 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:53:58,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:53:58,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:53:58,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.00 seconds 2025-02-15 09:53:58,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:53:58,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21058.74 MB 2025-02-15 09:53:58,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25167.45 MB 2025-02-15 09:53:58,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4108.71 MB 2025-02-15 09:53:58,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61220.06 MB 2025-02-15 09:53:58,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28699.53 MB 2025-02-15 09:53:58,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32520.54 MB 2025-02-15 09:53:58,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34153.99 MB 2025-02-15 09:53:58,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:53:58,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:53:58,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:53:58,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:53:58,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25167.45 MB 2025-02-15 09:53:58,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21813.53 MB 2025-02-15 09:53:58,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3353.93 MB 2025-02-15 09:53:58,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28699.53 MB 2025-02-15 09:53:58,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38273.02 MB 2025-02-15 09:53:58,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9573.50 MB 2025-02-15 09:53:58,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37546.82 MB 2025-02-15 09:54:00,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:54:00,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:54:00,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:54:00,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:00,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21813.53 MB 2025-02-15 09:54:00,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.37 MB 2025-02-15 09:54:00,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:54:00,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38273.02 MB 2025-02-15 09:54:00,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26004.68 MB 2025-02-15 09:54:00,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12268.34 MB 2025-02-15 09:54:00,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26322.92 MB 2025-02-15 09:54:00,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:54:00,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:54:00,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:54:00,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:00,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.37 MB 2025-02-15 09:54:00,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24233.90 MB 2025-02-15 09:54:00,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:54:00,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26004.68 MB 2025-02-15 09:54:00,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27892.12 MB 2025-02-15 09:54:00,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:54:00,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25651.33 MB 2025-02-15 09:54:00,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:54:00,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:54:00,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:54:00,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:00,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24233.90 MB 2025-02-15 09:54:00,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26475.76 MB 2025-02-15 09:54:00,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:54:00,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27892.12 MB 2025-02-15 09:54:00,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-15 09:54:00,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 09:54:00,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32021.09 MB 2025-02-15 09:54:00,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:54:00,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:54:00,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:54:00,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:00,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.37 MB 2025-02-15 09:54:00,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26475.76 MB 2025-02-15 09:54:00,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:54:00,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26004.68 MB 2025-02-15 09:54:00,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-15 09:54:00,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8258.58 MB 2025-02-15 09:54:00,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32021.09 MB 2025-02-15 09:54:01,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:54:01,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:54:01,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:54:01,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:01,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28010.35 MB 2025-02-15 09:54:01,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28777.35 MB 2025-02-15 09:54:01,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:54:01,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34263.27 MB 2025-02-15 09:54:01,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34676.41 MB 2025-02-15 09:54:01,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 09:54:01,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29485.14 MB 2025-02-15 09:54:01,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:54:01,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:54:01,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:54:01,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:01,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29190.24 MB 2025-02-15 09:54:01,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29419.37 MB 2025-02-15 09:54:01,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-15 09:54:01,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34676.41 MB 2025-02-15 09:54:01,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34676.41 MB 2025-02-15 09:54:01,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:54:01,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29643.11 MB 2025-02-15 09:54:01,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:54:01,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:54:01,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.43 seconds 2025-02-15 09:54:01,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:01,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17013.72 MB 2025-02-15 09:54:01,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29620.42 MB 2025-02-15 09:54:01,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12606.70 MB 2025-02-15 09:54:01,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61220.06 MB 2025-02-15 09:54:01,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34676.41 MB 2025-02-15 09:54:01,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26543.65 MB 2025-02-15 09:54:01,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29643.11 MB 2025-02-15 09:54:01,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:54:01,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:54:01,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:54:01,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:01,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29620.42 MB 2025-02-15 09:54:01,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22018.78 MB 2025-02-15 09:54:01,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7601.64 MB 2025-02-15 09:54:01,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34676.41 MB 2025-02-15 09:54:01,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34676.41 MB 2025-02-15 09:54:01,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:54:01,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32131.78 MB 2025-02-15 09:54:01,471 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 09:54:01,471 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:54:01,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:54:01,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:54:01,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:54:01,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:54:01,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22018.78 MB 2025-02-15 09:54:01,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30457.62 MB 2025-02-15 09:54:01,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 09:54:01,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34676.41 MB 2025-02-15 09:54:01,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43065.02 MB 2025-02-15 09:54:01,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 09:54:01,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30457.62 MB 2025-02-15 09:54:01,691 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 09:54:01,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:54:01,693 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:54:01,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:54:01,695 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:54:01,702 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:54:01,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:54:01,704 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:54:01,704 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:55:01,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:01,224 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:55:01,229 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:55:01,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:01,233 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1814, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:55:01,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:01,234 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1814, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:55:29,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:55:29,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:55:29,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.08 seconds 2025-02-15 09:55:29,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:29,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25608.95 MB 2025-02-15 09:55:29,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32028.59 MB 2025-02-15 09:55:29,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6419.64 MB 2025-02-15 09:55:29,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51453.62 MB 2025-02-15 09:55:29,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39397.10 MB 2025-02-15 09:55:29,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12056.53 MB 2025-02-15 09:55:29,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40969.12 MB 2025-02-15 09:55:29,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:55:29,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:55:29,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 09:55:29,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:29,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32028.59 MB 2025-02-15 09:55:29,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25208.27 MB 2025-02-15 09:55:29,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6820.32 MB 2025-02-15 09:55:29,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39397.10 MB 2025-02-15 09:55:29,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53198.45 MB 2025-02-15 09:55:29,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13801.36 MB 2025-02-15 09:55:29,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50652.37 MB 2025-02-15 09:55:31,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:55:31,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:55:31,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:55:31,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25208.27 MB 2025-02-15 09:55:31,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25739.11 MB 2025-02-15 09:55:31,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:55:31,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53198.45 MB 2025-02-15 09:55:31,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30337.40 MB 2025-02-15 09:55:31,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22861.05 MB 2025-02-15 09:55:31,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29717.66 MB 2025-02-15 09:55:31,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:55:31,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:55:31,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:55:31,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25739.11 MB 2025-02-15 09:55:31,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27628.65 MB 2025-02-15 09:55:31,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:55:31,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30337.40 MB 2025-02-15 09:55:31,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30337.40 MB 2025-02-15 09:55:31,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:55:31,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29046.08 MB 2025-02-15 09:55:31,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:55:31,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:55:31,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:55:31,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27628.65 MB 2025-02-15 09:55:31,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.50 MB 2025-02-15 09:55:31,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:55:31,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30337.40 MB 2025-02-15 09:55:31,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37652.27 MB 2025-02-15 09:55:31,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 09:55:31,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35414.78 MB 2025-02-15 09:55:31,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:55:31,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:55:31,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 09:55:31,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25739.11 MB 2025-02-15 09:55:31,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.50 MB 2025-02-15 09:55:31,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:55:31,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30337.40 MB 2025-02-15 09:55:31,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37652.27 MB 2025-02-15 09:55:31,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 09:55:31,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35414.78 MB 2025-02-15 09:55:31,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:55:31,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:55:31,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 09:55:31,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31404.04 MB 2025-02-15 09:55:31,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32171.05 MB 2025-02-15 09:55:31,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:55:31,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37652.27 MB 2025-02-15 09:55:31,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38067.50 MB 2025-02-15 09:55:31,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:55:31,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32878.84 MB 2025-02-15 09:55:31,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:55:31,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:55:31,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:55:31,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32583.94 MB 2025-02-15 09:55:31,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32812.11 MB 2025-02-15 09:55:31,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 09:55:31,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38067.50 MB 2025-02-15 09:55:31,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38067.50 MB 2025-02-15 09:55:31,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:55:31,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33017.09 MB 2025-02-15 09:55:31,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:55:31,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:55:31,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.60 seconds 2025-02-15 09:55:31,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:31,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.83 MB 2025-02-15 09:55:31,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33012.20 MB 2025-02-15 09:55:31,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13723.37 MB 2025-02-15 09:55:31,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51453.62 MB 2025-02-15 09:55:31,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38067.50 MB 2025-02-15 09:55:31,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13386.12 MB 2025-02-15 09:55:31,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33017.09 MB 2025-02-15 09:55:32,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:55:32,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:55:32,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:55:32,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:32,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33012.20 MB 2025-02-15 09:55:32,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24278.74 MB 2025-02-15 09:55:32,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8733.46 MB 2025-02-15 09:55:32,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38067.50 MB 2025-02-15 09:55:32,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38067.50 MB 2025-02-15 09:55:32,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:55:32,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35512.34 MB 2025-02-15 09:55:32,122 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 09:55:32,122 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:55:32,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:55:32,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:55:32,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:55:32,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:55:32,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.74 MB 2025-02-15 09:55:32,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32676.14 MB 2025-02-15 09:55:32,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 09:55:32,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38067.50 MB 2025-02-15 09:55:32,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42242.93 MB 2025-02-15 09:55:32,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 09:55:32,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32676.14 MB 2025-02-15 09:55:32,290 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 09:55:32,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:32,292 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:55:32,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:32,293 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:55:32,299 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:55:32,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:32,300 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:55:32,301 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:55:41,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:41,452 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:55:41,457 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:55:41,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:41,460 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1385, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:55:41,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:55:41,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1385, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:56:03,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:56:03,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:56:03,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.72 seconds 2025-02-15 09:56:03,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:03,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22619.61 MB 2025-02-15 09:56:03,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27521.05 MB 2025-02-15 09:56:03,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.44 MB 2025-02-15 09:56:03,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50593.79 MB 2025-02-15 09:56:03,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37960.55 MB 2025-02-15 09:56:03,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12633.24 MB 2025-02-15 09:56:03,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36394.33 MB 2025-02-15 09:56:03,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:56:03,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:56:03,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 09:56:03,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:03,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27521.05 MB 2025-02-15 09:56:03,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22978.03 MB 2025-02-15 09:56:03,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4543.01 MB 2025-02-15 09:56:03,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37960.55 MB 2025-02-15 09:56:03,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47674.56 MB 2025-02-15 09:56:03,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9714.01 MB 2025-02-15 09:56:03,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42129.46 MB 2025-02-15 09:56:05,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:56:05,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:56:05,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:56:05,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22978.03 MB 2025-02-15 09:56:05,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.88 MB 2025-02-15 09:56:05,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:56:05,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47674.56 MB 2025-02-15 09:56:05,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33057.41 MB 2025-02-15 09:56:05,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14617.15 MB 2025-02-15 09:56:05,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27487.42 MB 2025-02-15 09:56:05,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:56:05,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:56:05,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:56:05,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-15 09:56:05,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.41 MB 2025-02-15 09:56:05,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:56:05,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33057.41 MB 2025-02-15 09:56:05,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33057.41 MB 2025-02-15 09:56:05,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:05,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26815.84 MB 2025-02-15 09:56:05,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:56:05,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:56:05,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:56:05,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.41 MB 2025-02-15 09:56:05,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-15 09:56:05,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:56:05,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33057.41 MB 2025-02-15 09:56:05,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35888.56 MB 2025-02-15 09:56:05,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:56:05,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-15 09:56:05,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:56:05,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:56:05,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:56:05,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-15 09:56:05,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-15 09:56:05,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:56:05,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33057.41 MB 2025-02-15 09:56:05,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35888.56 MB 2025-02-15 09:56:05,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 09:56:05,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-15 09:56:05,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:56:05,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:56:05,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:56:05,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29173.81 MB 2025-02-15 09:56:05,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29940.81 MB 2025-02-15 09:56:05,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:56:05,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35888.56 MB 2025-02-15 09:56:05,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:56:05,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:56:05,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30648.60 MB 2025-02-15 09:56:05,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:56:05,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:56:05,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:56:05,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30353.70 MB 2025-02-15 09:56:05,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30582.56 MB 2025-02-15 09:56:05,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-15 09:56:05,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:56:05,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:56:05,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:05,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30827.63 MB 2025-02-15 09:56:05,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:56:05,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:56:05,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.14 seconds 2025-02-15 09:56:05,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17794.16 MB 2025-02-15 09:56:05,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30783.34 MB 2025-02-15 09:56:05,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.18 MB 2025-02-15 09:56:05,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50593.79 MB 2025-02-15 09:56:05,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:56:05,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14287.90 MB 2025-02-15 09:56:05,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30827.63 MB 2025-02-15 09:56:05,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:56:05,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:56:05,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:56:05,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30783.34 MB 2025-02-15 09:56:05,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22794.05 MB 2025-02-15 09:56:05,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7989.29 MB 2025-02-15 09:56:05,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:56:05,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36305.90 MB 2025-02-15 09:56:05,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:05,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33291.32 MB 2025-02-15 09:56:05,889 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 09:56:05,889 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 09:56:05,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:56:05,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:56:05,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:56:05,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:05,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22794.05 MB 2025-02-15 09:56:05,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31220.55 MB 2025-02-15 09:56:05,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 09:56:05,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36305.90 MB 2025-02-15 09:56:05,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44684.02 MB 2025-02-15 09:56:05,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-15 09:56:05,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31220.55 MB 2025-02-15 09:56:06,058 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 09:56:06,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:06,059 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:56:06,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:06,060 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:56:06,065 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:56:06,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:06,066 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:56:06,066 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 09:56:32,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:32,059 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:56:32,064 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:56:32,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:32,068 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:56:32,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:32,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:56:34,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:56:34,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:56:34,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-15 09:56:34,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:34,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-15 09:56:34,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-15 09:56:34,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 09:56:34,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57250.15 MB 2025-02-15 09:56:34,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21600.67 MB 2025-02-15 09:56:34,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35649.49 MB 2025-02-15 09:56:34,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-15 09:56:34,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:56:34,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:56:34,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:56:34,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:34,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-15 09:56:34,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-15 09:56:34,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-15 09:56:34,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21600.67 MB 2025-02-15 09:56:34,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21600.67 MB 2025-02-15 09:56:34,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:34,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16921.83 MB 2025-02-15 09:56:35,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:56:35,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:56:35,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-15 09:56:35,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-15 09:56:35,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-15 09:56:35,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 09:56:35,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21600.67 MB 2025-02-15 09:56:35,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21128.81 MB 2025-02-15 09:56:35,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 09:56:35,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19106.05 MB 2025-02-15 09:56:35,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:56:35,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:56:35,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 09:56:35,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 09:56:35,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-15 09:56:35,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 09:56:35,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21128.81 MB 2025-02-15 09:56:35,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21128.81 MB 2025-02-15 09:56:35,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:35,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-15 09:56:35,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:56:35,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:56:35,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:56:35,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-15 09:56:35,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 09:56:35,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 09:56:35,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21128.81 MB 2025-02-15 09:56:35,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21128.81 MB 2025-02-15 09:56:35,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:35,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-15 09:56:35,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:56:35,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:56:35,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 09:56:35,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 09:56:35,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 09:56:35,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 09:56:35,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21128.81 MB 2025-02-15 09:56:35,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21128.81 MB 2025-02-15 09:56:35,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:35,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-15 09:56:35,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:56:35,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:56:35,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 09:56:35,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-15 09:56:35,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-15 09:56:35,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 09:56:35,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21128.81 MB 2025-02-15 09:56:35,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21292.38 MB 2025-02-15 09:56:35,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 09:56:35,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18031.28 MB 2025-02-15 09:56:35,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:56:35,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:56:35,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:56:35,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-15 09:56:35,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18133.22 MB 2025-02-15 09:56:35,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-15 09:56:35,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21292.38 MB 2025-02-15 09:56:35,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21292.38 MB 2025-02-15 09:56:35,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:35,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18153.57 MB 2025-02-15 09:56:35,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:56:35,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:56:35,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-15 09:56:35,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-15 09:56:35,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18334.29 MB 2025-02-15 09:56:35,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4804.65 MB 2025-02-15 09:56:35,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57250.15 MB 2025-02-15 09:56:35,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21292.38 MB 2025-02-15 09:56:35,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35957.77 MB 2025-02-15 09:56:35,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18334.29 MB 2025-02-15 09:56:35,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:56:35,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:56:35,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:56:35,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18334.29 MB 2025-02-15 09:56:35,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.12 MB 2025-02-15 09:56:35,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -928.17 MB 2025-02-15 09:56:35,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21292.38 MB 2025-02-15 09:56:35,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21292.38 MB 2025-02-15 09:56:35,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:56:35,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19138.03 MB 2025-02-15 09:56:35,837 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:56:35,838 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:56:35,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:56:35,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:56:35,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 09:56:35,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:56:35,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.12 MB 2025-02-15 09:56:35,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25845.14 MB 2025-02-15 09:56:35,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:56:35,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21292.38 MB 2025-02-15 09:56:35,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29683.09 MB 2025-02-15 09:56:35,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 09:56:35,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.14 MB 2025-02-15 09:56:36,003 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:56:36,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:36,004 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:56:36,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:36,005 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:56:36,010 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:56:36,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:56:36,011 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:56:36,011 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 09:57:19,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:19,170 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:57:19,178 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:57:19,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:19,185 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 474, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:57:19,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:19,187 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 474, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:57:26,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:57:26,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:57:26,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.37 seconds 2025-02-15 09:57:26,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:26,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16271.61 MB 2025-02-15 09:57:26,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17949.07 MB 2025-02-15 09:57:26,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1677.46 MB 2025-02-15 09:57:26,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42268.10 MB 2025-02-15 09:57:26,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20652.75 MB 2025-02-15 09:57:26,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21615.35 MB 2025-02-15 09:57:26,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26875.45 MB 2025-02-15 09:57:26,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:57:26,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:57:26,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 09:57:26,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:26,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17949.07 MB 2025-02-15 09:57:26,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18243.08 MB 2025-02-15 09:57:26,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.00 MB 2025-02-15 09:57:26,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20652.75 MB 2025-02-15 09:57:26,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26061.31 MB 2025-02-15 09:57:26,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5408.56 MB 2025-02-15 09:57:26,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25455.05 MB 2025-02-15 09:57:28,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:57:28,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:57:28,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 09:57:28,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18243.08 MB 2025-02-15 09:57:28,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18773.92 MB 2025-02-15 09:57:28,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:57:28,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26061.31 MB 2025-02-15 09:57:28,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22777.17 MB 2025-02-15 09:57:28,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3284.14 MB 2025-02-15 09:57:28,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22754.39 MB 2025-02-15 09:57:28,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:57:28,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:57:28,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:57:28,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-15 09:57:28,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20663.45 MB 2025-02-15 09:57:28,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:57:28,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22777.17 MB 2025-02-15 09:57:28,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24664.60 MB 2025-02-15 09:57:28,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 09:57:28,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22080.88 MB 2025-02-15 09:57:28,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:57:28,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:57:28,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:57:28,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20663.45 MB 2025-02-15 09:57:28,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-15 09:57:28,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:57:28,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24664.60 MB 2025-02-15 09:57:28,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30798.77 MB 2025-02-15 09:57:28,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:57:28,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-15 09:57:28,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:57:28,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:57:28,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:57:28,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-15 09:57:28,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-15 09:57:28,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:57:28,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22777.17 MB 2025-02-15 09:57:28,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30798.77 MB 2025-02-15 09:57:28,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 09:57:28,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-15 09:57:28,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:57:28,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:57:28,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:57:28,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24438.85 MB 2025-02-15 09:57:28,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25205.85 MB 2025-02-15 09:57:28,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:57:28,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30798.77 MB 2025-02-15 09:57:28,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31216.11 MB 2025-02-15 09:57:28,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:57:28,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25913.64 MB 2025-02-15 09:57:28,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:57:28,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:57:28,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:57:28,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25618.74 MB 2025-02-15 09:57:28,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25847.65 MB 2025-02-15 09:57:28,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 09:57:28,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31216.11 MB 2025-02-15 09:57:28,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31216.11 MB 2025-02-15 09:57:28,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:57:28,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26069.11 MB 2025-02-15 09:57:28,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:57:28,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:57:28,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.74 seconds 2025-02-15 09:57:28,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:28,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14620.16 MB 2025-02-15 09:57:28,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.73 MB 2025-02-15 09:57:28,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11428.57 MB 2025-02-15 09:57:28,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42268.10 MB 2025-02-15 09:57:28,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31216.11 MB 2025-02-15 09:57:28,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11051.99 MB 2025-02-15 09:57:28,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26069.11 MB 2025-02-15 09:57:29,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:57:29,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:57:29,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:57:29,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:29,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.73 MB 2025-02-15 09:57:29,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19624.55 MB 2025-02-15 09:57:29,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6424.18 MB 2025-02-15 09:57:29,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31216.11 MB 2025-02-15 09:57:29,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31216.11 MB 2025-02-15 09:57:29,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:57:29,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28560.39 MB 2025-02-15 09:57:29,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 09:57:29,214 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 09:57:29,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:57:29,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:57:29,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:57:29,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:29,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19624.55 MB 2025-02-15 09:57:29,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28063.57 MB 2025-02-15 09:57:29,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 09:57:29,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31216.11 MB 2025-02-15 09:57:29,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41706.06 MB 2025-02-15 09:57:29,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 09:57:29,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28063.57 MB 2025-02-15 09:57:29,384 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 09:57:29,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:29,385 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:57:29,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:29,386 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:57:29,391 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:57:29,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:29,392 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:57:29,392 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 09:57:43,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:43,016 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:57:43,021 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:57:43,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:43,024 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 822, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:57:43,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:43,025 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 822, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:57:55,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:57:55,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:57:55,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.75 seconds 2025-02-15 09:57:55,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:55,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18696.53 MB 2025-02-15 09:57:55,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21605.55 MB 2025-02-15 09:57:55,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2909.01 MB 2025-02-15 09:57:55,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54291.07 MB 2025-02-15 09:57:55,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27080.52 MB 2025-02-15 09:57:55,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27210.55 MB 2025-02-15 09:57:55,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30432.83 MB 2025-02-15 09:57:55,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:57:55,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:57:55,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 09:57:55,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:55,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21605.55 MB 2025-02-15 09:57:55,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20052.22 MB 2025-02-15 09:57:55,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1553.33 MB 2025-02-15 09:57:55,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27080.52 MB 2025-02-15 09:57:55,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 09:57:55,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7631.54 MB 2025-02-15 09:57:55,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31338.50 MB 2025-02-15 09:57:57,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:57:57,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:57:57,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 09:57:57,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:57,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20052.22 MB 2025-02-15 09:57:57,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20583.06 MB 2025-02-15 09:57:57,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:57:57,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-15 09:57:57,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26294.09 MB 2025-02-15 09:57:57,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8417.97 MB 2025-02-15 09:57:57,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24561.61 MB 2025-02-15 09:57:57,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:57:57,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:57:57,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:57:57,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:57,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.06 MB 2025-02-15 09:57:57,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22472.60 MB 2025-02-15 09:57:57,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:57:57,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26294.09 MB 2025-02-15 09:57:57,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26294.09 MB 2025-02-15 09:57:57,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:57:57,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23890.03 MB 2025-02-15 09:57:57,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:57:57,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:57:57,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 09:57:57,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:57,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22472.60 MB 2025-02-15 09:57:57,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24714.45 MB 2025-02-15 09:57:57,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:57:57,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26294.09 MB 2025-02-15 09:57:57,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32428.26 MB 2025-02-15 09:57:57,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:57:57,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30258.73 MB 2025-02-15 09:57:57,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:57:57,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:57:57,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:57:57,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:57,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.06 MB 2025-02-15 09:57:57,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24714.45 MB 2025-02-15 09:57:57,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:57:57,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26294.09 MB 2025-02-15 09:57:57,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32428.26 MB 2025-02-15 09:57:57,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:57:57,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30258.73 MB 2025-02-15 09:57:58,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:57:58,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:57:58,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:57:58,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:58,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26247.99 MB 2025-02-15 09:57:58,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27015.00 MB 2025-02-15 09:57:58,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:57:58,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32428.26 MB 2025-02-15 09:57:58,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 09:57:58,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 09:57:58,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27722.79 MB 2025-02-15 09:57:58,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:57:58,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:57:58,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:57:58,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:58,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27427.89 MB 2025-02-15 09:57:58,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27654.86 MB 2025-02-15 09:57:58,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.97 MB 2025-02-15 09:57:58,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-15 09:57:58,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 09:57:58,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:57:58,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27851.58 MB 2025-02-15 09:57:58,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:57:58,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:57:58,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.14 seconds 2025-02-15 09:57:58,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:58,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15832.62 MB 2025-02-15 09:57:58,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27855.76 MB 2025-02-15 09:57:58,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12023.14 MB 2025-02-15 09:57:58,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54291.07 MB 2025-02-15 09:57:58,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 09:57:58,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21447.57 MB 2025-02-15 09:57:58,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27855.76 MB 2025-02-15 09:57:58,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:57:58,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:57:58,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:57:58,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:58,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27855.76 MB 2025-02-15 09:57:58,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20834.34 MB 2025-02-15 09:57:58,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7021.41 MB 2025-02-15 09:57:58,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-15 09:57:58,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 09:57:58,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:57:58,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30365.27 MB 2025-02-15 09:57:58,453 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 09:57:58,454 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:57:58,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:57:58,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:57:58,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:57:58,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:57:58,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20834.34 MB 2025-02-15 09:57:58,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29265.81 MB 2025-02-15 09:57:58,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 09:57:58,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-15 09:57:58,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41227.91 MB 2025-02-15 09:57:58,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 09:57:58,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29265.81 MB 2025-02-15 09:57:58,620 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 09:57:58,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:58,621 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:57:58,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:58,623 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:57:58,628 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:57:58,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:57:58,630 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:57:58,630 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:58:48,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:48,634 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:58:48,638 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:58:48,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:48,642 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:58:48,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:48,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:58:52,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:58:52,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:58:52,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.61 seconds 2025-02-15 09:58:52,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:52,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-15 09:58:52,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15427.37 MB 2025-02-15 09:58:52,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-15 09:58:52,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49612.32 MB 2025-02-15 09:58:52,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 09:58:52,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29855.06 MB 2025-02-15 09:58:52,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.12 MB 2025-02-15 09:58:52,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:58:52,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:58:52,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:58:52,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:52,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15427.37 MB 2025-02-15 09:58:52,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15745.03 MB 2025-02-15 09:58:52,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.66 MB 2025-02-15 09:58:52,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 09:58:52,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20543.70 MB 2025-02-15 09:58:52,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 786.43 MB 2025-02-15 09:58:52,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18546.38 MB 2025-02-15 09:58:53,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:58:53,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:58:53,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-15 09:58:53,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15745.03 MB 2025-02-15 09:58:53,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16039.65 MB 2025-02-15 09:58:53,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-15 09:58:53,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20543.70 MB 2025-02-15 09:58:53,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 09:58:53,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 09:58:53,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20000.66 MB 2025-02-15 09:58:53,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:58:53,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:58:53,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:58:53,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16039.65 MB 2025-02-15 09:58:53,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17088.09 MB 2025-02-15 09:58:53,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-15 09:58:53,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 09:58:53,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 09:58:53,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:58:53,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.76 MB 2025-02-15 09:58:53,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:58:53,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:58:53,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 09:58:53,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17088.09 MB 2025-02-15 09:58:53,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18332.34 MB 2025-02-15 09:58:53,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.26 MB 2025-02-15 09:58:53,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 09:58:53,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-15 09:58:53,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2621.44 MB 2025-02-15 09:58:53,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21412.01 MB 2025-02-15 09:58:53,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:58:53,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:58:53,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 09:58:53,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16039.65 MB 2025-02-15 09:58:53,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18332.34 MB 2025-02-15 09:58:53,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2292.70 MB 2025-02-15 09:58:53,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 09:58:53,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-15 09:58:53,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2621.44 MB 2025-02-15 09:58:53,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21412.01 MB 2025-02-15 09:58:53,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:58:53,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:58:53,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:58:53,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19183.46 MB 2025-02-15 09:58:53,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19609.67 MB 2025-02-15 09:58:53,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 426.21 MB 2025-02-15 09:58:53,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22693.28 MB 2025-02-15 09:58:53,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22923.97 MB 2025-02-15 09:58:53,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 230.69 MB 2025-02-15 09:58:53,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20002.49 MB 2025-02-15 09:58:53,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:58:53,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:58:53,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:58:53,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19838.83 MB 2025-02-15 09:58:53,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20060.66 MB 2025-02-15 09:58:53,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.83 MB 2025-02-15 09:58:53,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22923.97 MB 2025-02-15 09:58:53,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22923.97 MB 2025-02-15 09:58:53,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:58:53,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20132.09 MB 2025-02-15 09:58:53,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:58:53,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:58:53,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.94 seconds 2025-02-15 09:58:53,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13783.98 MB 2025-02-15 09:58:53,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20261.61 MB 2025-02-15 09:58:53,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6477.63 MB 2025-02-15 09:58:53,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49612.32 MB 2025-02-15 09:58:53,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22923.97 MB 2025-02-15 09:58:53,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26688.36 MB 2025-02-15 09:58:53,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20261.61 MB 2025-02-15 09:58:53,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:58:53,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:58:53,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 09:58:53,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14934.77 MB 2025-02-15 09:58:53,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17946.96 MB 2025-02-15 09:58:53,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.19 MB 2025-02-15 09:58:53,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22923.97 MB 2025-02-15 09:58:53,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22923.97 MB 2025-02-15 09:58:53,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:58:53,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18248.14 MB 2025-02-15 09:58:53,865 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 09:58:53,866 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:58:53,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:58:53,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:58:53,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:58:53,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:58:53,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17946.96 MB 2025-02-15 09:58:53,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26381.57 MB 2025-02-15 09:58:53,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 09:58:53,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22923.97 MB 2025-02-15 09:58:53,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31308.38 MB 2025-02-15 09:58:53,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 09:58:53,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26381.57 MB 2025-02-15 09:58:54,031 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 09:58:54,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:54,032 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:58:54,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:54,033 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:58:54,038 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:58:54,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:58:54,039 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:58:54,039 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 09:59:04,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:04,219 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 09:59:04,224 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 09:59:04,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:04,227 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1025, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 09:59:04,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:04,228 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1025, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 09:59:20,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 09:59:20,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 09:59:20,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.88 seconds 2025-02-15 09:59:20,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:20,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20111.07 MB 2025-02-15 09:59:20,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23739.14 MB 2025-02-15 09:59:20,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3628.07 MB 2025-02-15 09:59:20,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39692.80 MB 2025-02-15 09:59:20,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26396.85 MB 2025-02-15 09:59:20,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13295.94 MB 2025-02-15 09:59:20,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32754.14 MB 2025-02-15 09:59:20,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 09:59:20,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 09:59:20,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 09:59:20,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:20,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23739.14 MB 2025-02-15 09:59:20,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21107.55 MB 2025-02-15 09:59:20,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2631.59 MB 2025-02-15 09:59:20,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26396.85 MB 2025-02-15 09:59:20,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 09:59:20,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9275.70 MB 2025-02-15 09:59:20,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35114.81 MB 2025-02-15 09:59:22,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 09:59:22,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 09:59:22,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 09:59:22,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21107.55 MB 2025-02-15 09:59:22,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21638.40 MB 2025-02-15 09:59:22,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 09:59:22,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35672.56 MB 2025-02-15 09:59:22,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22793.95 MB 2025-02-15 09:59:22,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12878.61 MB 2025-02-15 09:59:22,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25619.02 MB 2025-02-15 09:59:22,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 09:59:22,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 09:59:22,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 09:59:22,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21638.40 MB 2025-02-15 09:59:22,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23527.93 MB 2025-02-15 09:59:22,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 09:59:22,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 09:59:22,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26568.82 MB 2025-02-15 09:59:22,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 09:59:22,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24945.36 MB 2025-02-15 09:59:22,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 09:59:22,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 09:59:22,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 09:59:22,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23527.93 MB 2025-02-15 09:59:22,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25769.79 MB 2025-02-15 09:59:22,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 09:59:22,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26568.82 MB 2025-02-15 09:59:22,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32702.99 MB 2025-02-15 09:59:22,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 09:59:22,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31314.07 MB 2025-02-15 09:59:22,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 09:59:22,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 09:59:22,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 09:59:22,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21638.40 MB 2025-02-15 09:59:22,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25769.79 MB 2025-02-15 09:59:22,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 09:59:22,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22793.95 MB 2025-02-15 09:59:22,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32702.99 MB 2025-02-15 09:59:22,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-15 09:59:22,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31314.07 MB 2025-02-15 09:59:22,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 09:59:22,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 09:59:22,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 09:59:22,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27303.33 MB 2025-02-15 09:59:22,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28070.33 MB 2025-02-15 09:59:22,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 09:59:22,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32702.99 MB 2025-02-15 09:59:22,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33120.32 MB 2025-02-15 09:59:22,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 09:59:22,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28778.12 MB 2025-02-15 09:59:22,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 09:59:22,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 09:59:22,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:59:22,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28483.22 MB 2025-02-15 09:59:22,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28710.68 MB 2025-02-15 09:59:22,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.46 MB 2025-02-15 09:59:22,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33120.32 MB 2025-02-15 09:59:22,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33120.32 MB 2025-02-15 09:59:22,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:59:22,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28940.52 MB 2025-02-15 09:59:22,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 09:59:22,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 09:59:22,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.32 seconds 2025-02-15 09:59:22,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16539.89 MB 2025-02-15 09:59:22,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28911.53 MB 2025-02-15 09:59:22,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12371.64 MB 2025-02-15 09:59:22,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39692.80 MB 2025-02-15 09:59:22,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33120.32 MB 2025-02-15 09:59:22,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6572.47 MB 2025-02-15 09:59:22,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28940.52 MB 2025-02-15 09:59:22,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 09:59:22,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 09:59:22,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 09:59:22,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28911.53 MB 2025-02-15 09:59:22,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21536.22 MB 2025-02-15 09:59:22,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7375.32 MB 2025-02-15 09:59:22,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33120.32 MB 2025-02-15 09:59:22,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33120.32 MB 2025-02-15 09:59:22,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 09:59:22,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31416.44 MB 2025-02-15 09:59:22,838 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 09:59:22,838 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 09:59:22,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 09:59:22,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 09:59:22,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 09:59:22,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 09:59:22,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21536.22 MB 2025-02-15 09:59:22,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29952.82 MB 2025-02-15 09:59:22,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 09:59:22,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33120.32 MB 2025-02-15 09:59:22,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41487.96 MB 2025-02-15 09:59:22,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 09:59:22,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29952.82 MB 2025-02-15 09:59:23,003 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 09:59:23,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:23,005 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 09:59:23,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:23,007 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 09:59:23,012 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 09:59:23,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 09:59:23,013 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 09:59:23,013 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:00:08,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:08,050 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:00:08,058 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:00:08,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:08,064 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:00:08,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:08,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:00:10,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:00:10,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:00:10,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.93 seconds 2025-02-15 10:00:10,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:10,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14257.82 MB 2025-02-15 10:00:10,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14912.52 MB 2025-02-15 10:00:10,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-15 10:00:10,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49855.59 MB 2025-02-15 10:00:10,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20063.45 MB 2025-02-15 10:00:10,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29792.14 MB 2025-02-15 10:00:10,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23730.17 MB 2025-02-15 10:00:11,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:00:11,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:00:11,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:00:11,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:11,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14912.52 MB 2025-02-15 10:00:11,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.63 MB 2025-02-15 10:00:11,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 289.11 MB 2025-02-15 10:00:11,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20063.45 MB 2025-02-15 10:00:11,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20063.45 MB 2025-02-15 10:00:11,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:00:11,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17454.92 MB 2025-02-15 10:00:11,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:00:11,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:00:11,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.88 seconds 2025-02-15 10:00:11,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:11,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15201.63 MB 2025-02-15 10:00:11,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15441.84 MB 2025-02-15 10:00:11,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.21 MB 2025-02-15 10:00:11,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20063.45 MB 2025-02-15 10:00:11,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 10:00:11,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1598.03 MB 2025-02-15 10:00:11,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19372.32 MB 2025-02-15 10:00:11,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:00:11,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:00:11,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:00:11,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:11,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15441.77 MB 2025-02-15 10:00:11,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16297.36 MB 2025-02-15 10:00:11,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 855.59 MB 2025-02-15 10:00:11,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 10:00:11,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18893.24 MB 2025-02-15 10:00:11,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 427.82 MB 2025-02-15 10:00:11,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16938.76 MB 2025-02-15 10:00:11,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:00:11,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:00:11,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:00:11,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:11,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16297.36 MB 2025-02-15 10:00:11,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17311.84 MB 2025-02-15 10:00:11,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.48 MB 2025-02-15 10:00:11,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18893.24 MB 2025-02-15 10:00:11,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21246.25 MB 2025-02-15 10:00:11,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2353.00 MB 2025-02-15 10:00:11,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19824.13 MB 2025-02-15 10:00:11,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:00:11,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:00:11,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:00:11,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:11,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15441.77 MB 2025-02-15 10:00:11,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17311.84 MB 2025-02-15 10:00:11,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1870.07 MB 2025-02-15 10:00:11,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 10:00:11,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21246.25 MB 2025-02-15 10:00:11,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2780.82 MB 2025-02-15 10:00:11,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19824.13 MB 2025-02-15 10:00:12,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:00:12,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:00:12,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:00:12,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:12,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18005.77 MB 2025-02-15 10:00:12,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18353.23 MB 2025-02-15 10:00:12,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.46 MB 2025-02-15 10:00:12,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21246.25 MB 2025-02-15 10:00:12,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 10:00:12,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-15 10:00:12,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18678.12 MB 2025-02-15 10:00:12,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:00:12,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:00:12,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:00:12,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:12,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.07 MB 2025-02-15 10:00:12,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18756.92 MB 2025-02-15 10:00:12,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.85 MB 2025-02-15 10:00:12,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 10:00:12,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 10:00:12,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:00:12,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18795.28 MB 2025-02-15 10:00:12,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:00:12,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:00:12,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.02 seconds 2025-02-15 10:00:12,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:12,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13613.26 MB 2025-02-15 10:00:12,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18957.65 MB 2025-02-15 10:00:12,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5344.38 MB 2025-02-15 10:00:12,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49855.59 MB 2025-02-15 10:00:12,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 10:00:12,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28422.70 MB 2025-02-15 10:00:12,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18957.65 MB 2025-02-15 10:00:12,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:00:12,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:00:12,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:00:12,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:12,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18957.65 MB 2025-02-15 10:00:12,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17579.19 MB 2025-02-15 10:00:12,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1378.46 MB 2025-02-15 10:00:12,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 10:00:12,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 10:00:12,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:00:12,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18957.65 MB 2025-02-15 10:00:12,371 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 10:00:12,371 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:00:12,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:00:12,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:00:12,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:00:12,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:12,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17579.19 MB 2025-02-15 10:00:12,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26004.14 MB 2025-02-15 10:00:12,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 10:00:12,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 10:00:12,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31903.97 MB 2025-02-15 10:00:12,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 10:00:12,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26004.14 MB 2025-02-15 10:00:12,537 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 10:00:12,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:12,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:00:12,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:12,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:00:12,544 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:00:12,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:12,545 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:00:12,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:00:35,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:35,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:00:35,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:00:35,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:35,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:00:35,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:35,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:00:47,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:00:47,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:00:47,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.43 seconds 2025-02-15 10:00:47,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:47,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18585.04 MB 2025-02-15 10:00:47,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21437.43 MB 2025-02-15 10:00:47,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-15 10:00:47,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40280.00 MB 2025-02-15 10:00:47,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25614.61 MB 2025-02-15 10:00:47,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14665.38 MB 2025-02-15 10:00:47,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30322.15 MB 2025-02-15 10:00:47,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:00:47,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:00:47,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 10:00:47,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:47,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21437.43 MB 2025-02-15 10:00:47,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19967.99 MB 2025-02-15 10:00:47,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1469.44 MB 2025-02-15 10:00:47,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25614.61 MB 2025-02-15 10:00:47,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-15 10:00:47,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6981.42 MB 2025-02-15 10:00:47,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31226.49 MB 2025-02-15 10:00:49,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:00:49,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:00:49,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:00:49,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:49,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19967.99 MB 2025-02-15 10:00:49,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20498.84 MB 2025-02-15 10:00:49,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:00:49,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32596.03 MB 2025-02-15 10:00:49,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24175.97 MB 2025-02-15 10:00:49,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8420.07 MB 2025-02-15 10:00:49,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24478.42 MB 2025-02-15 10:00:49,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:00:49,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:00:49,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:00:49,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:49,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-15 10:00:49,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22388.37 MB 2025-02-15 10:00:49,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:00:49,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 10:00:49,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26063.41 MB 2025-02-15 10:00:49,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:00:49,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23805.80 MB 2025-02-15 10:00:49,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:00:49,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:00:49,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 10:00:49,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:49,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22388.37 MB 2025-02-15 10:00:49,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-15 10:00:49,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:00:49,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26063.41 MB 2025-02-15 10:00:49,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31725.72 MB 2025-02-15 10:00:49,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:00:49,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-15 10:00:49,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:00:49,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:00:49,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:00:49,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:49,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-15 10:00:49,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-15 10:00:49,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:00:49,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24175.97 MB 2025-02-15 10:00:49,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31725.72 MB 2025-02-15 10:00:49,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 10:00:49,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-15 10:00:50,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:00:50,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:00:50,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:00:50,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:50,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26163.77 MB 2025-02-15 10:00:50,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26930.77 MB 2025-02-15 10:00:50,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:00:50,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31725.72 MB 2025-02-15 10:00:50,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32143.05 MB 2025-02-15 10:00:50,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:00:50,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27638.56 MB 2025-02-15 10:00:50,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:00:50,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:00:50,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:00:50,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:50,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27343.66 MB 2025-02-15 10:00:50,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27571.86 MB 2025-02-15 10:00:50,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-15 10:00:50,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32143.05 MB 2025-02-15 10:00:50,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32143.05 MB 2025-02-15 10:00:50,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:00:50,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27778.07 MB 2025-02-15 10:00:50,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:00:50,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:00:50,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.83 seconds 2025-02-15 10:00:50,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:50,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.88 MB 2025-02-15 10:00:50,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27772.71 MB 2025-02-15 10:00:50,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11995.83 MB 2025-02-15 10:00:50,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40280.00 MB 2025-02-15 10:00:50,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32143.05 MB 2025-02-15 10:00:50,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8136.95 MB 2025-02-15 10:00:50,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27778.07 MB 2025-02-15 10:00:50,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:00:50,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:00:50,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:00:50,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:50,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27772.71 MB 2025-02-15 10:00:50,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20772.49 MB 2025-02-15 10:00:50,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7000.22 MB 2025-02-15 10:00:50,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32143.05 MB 2025-02-15 10:00:50,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32143.05 MB 2025-02-15 10:00:50,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:00:50,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30277.00 MB 2025-02-15 10:00:50,359 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 10:00:50,359 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:00:50,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:00:50,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:00:50,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:00:50,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:00:50,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20772.49 MB 2025-02-15 10:00:50,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29186.47 MB 2025-02-15 10:00:50,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 10:00:50,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32143.05 MB 2025-02-15 10:00:50,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40508.59 MB 2025-02-15 10:00:50,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 10:00:50,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29186.47 MB 2025-02-15 10:00:50,525 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 10:00:50,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:50,527 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:00:50,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:50,528 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:00:50,532 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:00:50,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:00:50,533 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:00:50,533 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:01:33,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:33,594 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:01:33,599 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:01:33,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:33,603 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:01:33,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:33,604 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:01:40,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:01:40,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:01:40,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.34 seconds 2025-02-15 10:01:40,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:40,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.65 MB 2025-02-15 10:01:40,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.57 MB 2025-02-15 10:01:40,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.92 MB 2025-02-15 10:01:40,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53055.85 MB 2025-02-15 10:01:40,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20254.29 MB 2025-02-15 10:01:40,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32801.55 MB 2025-02-15 10:01:40,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26868.48 MB 2025-02-15 10:01:40,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:01:40,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:01:40,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:01:40,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:40,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.57 MB 2025-02-15 10:01:40,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18237.88 MB 2025-02-15 10:01:40,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.31 MB 2025-02-15 10:01:40,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20254.29 MB 2025-02-15 10:01:40,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25572.67 MB 2025-02-15 10:01:40,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5318.38 MB 2025-02-15 10:01:40,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25352.85 MB 2025-02-15 10:01:42,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:01:42,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:01:42,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:01:42,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:42,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18237.88 MB 2025-02-15 10:01:42,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18768.72 MB 2025-02-15 10:01:42,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:01:42,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25572.67 MB 2025-02-15 10:01:42,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21434.99 MB 2025-02-15 10:01:42,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4137.68 MB 2025-02-15 10:01:42,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22748.31 MB 2025-02-15 10:01:42,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:01:42,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:01:42,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:01:42,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:42,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-15 10:01:42,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20658.25 MB 2025-02-15 10:01:42,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:01:42,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 10:01:42,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24266.15 MB 2025-02-15 10:01:42,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:01:42,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22075.68 MB 2025-02-15 10:01:43,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:01:43,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:01:43,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:01:43,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20658.25 MB 2025-02-15 10:01:43,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-15 10:01:43,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:01:43,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24266.15 MB 2025-02-15 10:01:43,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 10:01:43,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:01:43,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-15 10:01:43,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:01:43,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:01:43,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:01:43,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-15 10:01:43,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-15 10:01:43,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:01:43,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 10:01:43,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 10:01:43,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:01:43,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-15 10:01:43,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:01:43,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:01:43,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:01:43,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24433.65 MB 2025-02-15 10:01:43,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25200.65 MB 2025-02-15 10:01:43,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:01:43,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-15 10:01:43,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:01:43,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:01:43,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.44 MB 2025-02-15 10:01:43,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:01:43,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:01:43,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:01:43,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25613.54 MB 2025-02-15 10:01:43,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25847.10 MB 2025-02-15 10:01:43,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.56 MB 2025-02-15 10:01:43,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:01:43,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:01:43,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:01:43,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26028.69 MB 2025-02-15 10:01:43,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:01:43,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:01:43,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.74 seconds 2025-02-15 10:01:43,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14616.68 MB 2025-02-15 10:01:43,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.17 MB 2025-02-15 10:01:43,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11431.50 MB 2025-02-15 10:01:43,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53055.85 MB 2025-02-15 10:01:43,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:01:43,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22240.30 MB 2025-02-15 10:01:43,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26048.17 MB 2025-02-15 10:01:43,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:01:43,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:01:43,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:01:43,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.17 MB 2025-02-15 10:01:43,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19621.07 MB 2025-02-15 10:01:43,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6427.11 MB 2025-02-15 10:01:43,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:01:43,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:01:43,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:01:43,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28559.84 MB 2025-02-15 10:01:43,637 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:01:43,638 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:01:43,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:01:43,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:01:43,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 10:01:43,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:01:43,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.07 MB 2025-02-15 10:01:43,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28060.09 MB 2025-02-15 10:01:43,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:01:43,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:01:43,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 10:01:43,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:01:43,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28060.09 MB 2025-02-15 10:01:43,804 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:01:43,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:43,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:01:43,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:43,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:01:43,811 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:01:43,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:01:43,812 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:01:43,812 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:02:42,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:02:42,602 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:02:42,607 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:02:42,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:02:42,611 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:02:42,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:02:42,612 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:02:59,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:02:59,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:02:59,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.62 seconds 2025-02-15 10:02:59,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:02:59,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-15 10:02:59,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-15 10:02:59,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-15 10:02:59,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 10:02:59,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26575.11 MB 2025-02-15 10:02:59,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27315.40 MB 2025-02-15 10:02:59,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33095.58 MB 2025-02-15 10:02:59,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:02:59,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:02:59,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:02:59,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:02:59,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-15 10:02:59,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21362.29 MB 2025-02-15 10:02:59,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2891.05 MB 2025-02-15 10:02:59,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26575.11 MB 2025-02-15 10:02:59,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34152.12 MB 2025-02-15 10:02:59,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7577.01 MB 2025-02-15 10:02:59,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33930.61 MB 2025-02-15 10:03:01,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:03:01,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:03:01,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:03:01,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.29 MB 2025-02-15 10:03:01,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21893.13 MB 2025-02-15 10:03:01,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:03:01,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34152.12 MB 2025-02-15 10:03:01,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 10:03:01,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9254.73 MB 2025-02-15 10:03:01,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25872.72 MB 2025-02-15 10:03:01,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:03:01,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:03:01,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:03:01,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-15 10:03:01,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23782.67 MB 2025-02-15 10:03:01,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:03:01,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 10:03:01,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27728.54 MB 2025-02-15 10:03:01,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:03:01,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25200.10 MB 2025-02-15 10:03:01,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:03:01,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:03:01,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:03:01,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23782.67 MB 2025-02-15 10:03:01,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-15 10:03:01,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:03:01,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27728.54 MB 2025-02-15 10:03:01,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33862.71 MB 2025-02-15 10:03:01,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:03:01,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-15 10:03:01,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:03:01,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:03:01,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:03:01,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-15 10:03:01,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-15 10:03:01,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:03:01,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 10:03:01,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33862.71 MB 2025-02-15 10:03:01,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:03:01,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-15 10:03:01,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:03:01,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:03:01,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:03:01,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27558.06 MB 2025-02-15 10:03:01,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28325.07 MB 2025-02-15 10:03:01,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:03:01,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33862.71 MB 2025-02-15 10:03:01,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 10:03:01,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:03:01,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29032.86 MB 2025-02-15 10:03:01,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:03:01,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:03:01,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:03:01,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28737.96 MB 2025-02-15 10:03:01,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28968.07 MB 2025-02-15 10:03:01,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.11 MB 2025-02-15 10:03:01,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 10:03:01,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 10:03:01,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:03:01,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29183.84 MB 2025-02-15 10:03:01,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:03:01,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:03:01,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.09 seconds 2025-02-15 10:03:01,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-15 10:03:01,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29169.14 MB 2025-02-15 10:03:01,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12458.53 MB 2025-02-15 10:03:01,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 10:03:01,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 10:03:01,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-15 10:03:01,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29183.84 MB 2025-02-15 10:03:01,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:03:01,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:03:01,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:03:01,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:01,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29169.14 MB 2025-02-15 10:03:01,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.00 MB 2025-02-15 10:03:01,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7454.14 MB 2025-02-15 10:03:01,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 10:03:01,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 10:03:01,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:03:01,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31680.81 MB 2025-02-15 10:03:01,998 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:03:01,998 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:03:02,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:03:02,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:03:02,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:03:02,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:03:02,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.00 MB 2025-02-15 10:03:02,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30154.02 MB 2025-02-15 10:03:02,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:03:02,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 10:03:02,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44767.90 MB 2025-02-15 10:03:02,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:03:02,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.02 MB 2025-02-15 10:03:02,168 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:03:02,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:03:02,170 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:03:02,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:03:02,171 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:03:02,176 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:03:02,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:03:02,177 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:03:02,177 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:04:03,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:03,631 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:04:03,636 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:04:03,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:03,641 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1846, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:04:03,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:03,642 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1846, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:04:32,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:04:32,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:04:32,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.58 seconds 2025-02-15 10:04:32,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:32,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25831.93 MB 2025-02-15 10:04:32,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32364.82 MB 2025-02-15 10:04:32,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6532.89 MB 2025-02-15 10:04:32,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57352.91 MB 2025-02-15 10:04:32,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-15 10:04:32,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17559.45 MB 2025-02-15 10:04:32,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41192.10 MB 2025-02-15 10:04:32,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:04:32,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:04:32,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:04:32,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:32,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32364.82 MB 2025-02-15 10:04:32,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25374.63 MB 2025-02-15 10:04:32,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6990.19 MB 2025-02-15 10:04:32,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39793.46 MB 2025-02-15 10:04:32,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41209.04 MB 2025-02-15 10:04:32,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 10:04:32,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39939.59 MB 2025-02-15 10:04:34,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:04:34,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:04:34,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 10:04:34,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25374.63 MB 2025-02-15 10:04:34,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25905.47 MB 2025-02-15 10:04:34,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:04:34,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41209.04 MB 2025-02-15 10:04:34,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:04:34,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6534.73 MB 2025-02-15 10:04:34,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29884.02 MB 2025-02-15 10:04:34,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:04:34,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:04:34,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:04:34,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25905.47 MB 2025-02-15 10:04:34,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27795.00 MB 2025-02-15 10:04:34,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:04:34,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:04:34,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:04:34,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:04:34,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29212.43 MB 2025-02-15 10:04:34,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:04:34,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:04:34,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:04:34,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27795.00 MB 2025-02-15 10:04:34,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-15 10:04:34,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:04:34,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:04:34,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37977.33 MB 2025-02-15 10:04:34,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 10:04:34,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35581.14 MB 2025-02-15 10:04:34,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:04:34,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:04:34,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:04:34,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25905.47 MB 2025-02-15 10:04:34,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-15 10:04:34,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:04:34,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:04:34,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37977.33 MB 2025-02-15 10:04:34,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 10:04:34,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35581.14 MB 2025-02-15 10:04:34,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:04:34,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:04:34,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 10:04:34,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31570.40 MB 2025-02-15 10:04:34,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32337.40 MB 2025-02-15 10:04:34,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:04:34,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37977.33 MB 2025-02-15 10:04:34,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-15 10:04:34,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:04:34,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33045.19 MB 2025-02-15 10:04:34,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:04:34,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:04:34,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:04:34,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32750.29 MB 2025-02-15 10:04:34,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32984.23 MB 2025-02-15 10:04:34,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.94 MB 2025-02-15 10:04:34,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-15 10:04:34,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-15 10:04:34,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:04:34,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33166.32 MB 2025-02-15 10:04:34,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:04:34,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:04:34,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.15 seconds 2025-02-15 10:04:34,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:34,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19400.32 MB 2025-02-15 10:04:34,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33185.30 MB 2025-02-15 10:04:34,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13784.99 MB 2025-02-15 10:04:34,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57352.91 MB 2025-02-15 10:04:34,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-15 10:04:34,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18962.45 MB 2025-02-15 10:04:34,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33185.30 MB 2025-02-15 10:04:35,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:04:35,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:04:35,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:04:35,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:35,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33185.30 MB 2025-02-15 10:04:35,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24404.71 MB 2025-02-15 10:04:35,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8780.60 MB 2025-02-15 10:04:35,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-15 10:04:35,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-15 10:04:35,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:04:35,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35696.97 MB 2025-02-15 10:04:35,084 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:04:35,085 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:04:35,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:04:35,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:04:35,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:04:35,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:04:35,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24404.71 MB 2025-02-15 10:04:35,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32843.73 MB 2025-02-15 10:04:35,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:04:35,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-15 10:04:35,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42586.87 MB 2025-02-15 10:04:35,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 10:04:35,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32843.73 MB 2025-02-15 10:04:35,248 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:04:35,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:35,250 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:04:35,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:35,251 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:04:35,256 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:04:35,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:35,257 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:04:35,257 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:04:57,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:57,521 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:04:57,526 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:04:57,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:57,529 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:04:57,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:04:57,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:05:16,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:05:16,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:05:16,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.80 seconds 2025-02-15 10:05:16,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:16,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-15 10:05:16,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-15 10:05:16,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-15 10:05:16,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55171.87 MB 2025-02-15 10:05:16,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33342.62 MB 2025-02-15 10:05:16,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21829.26 MB 2025-02-15 10:05:16,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.46 MB 2025-02-15 10:05:16,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:05:16,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:05:16,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:05:16,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:16,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.80 MB 2025-02-15 10:05:16,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.07 MB 2025-02-15 10:05:16,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3608.73 MB 2025-02-15 10:05:16,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33342.62 MB 2025-02-15 10:05:16,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41584.43 MB 2025-02-15 10:05:16,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8241.81 MB 2025-02-15 10:05:16,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38135.18 MB 2025-02-15 10:05:18,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:05:18,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:05:18,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:05:18,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.07 MB 2025-02-15 10:05:18,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22593.91 MB 2025-02-15 10:05:18,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:05:18,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41584.43 MB 2025-02-15 10:05:18,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29062.33 MB 2025-02-15 10:05:18,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12522.09 MB 2025-02-15 10:05:18,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26572.45 MB 2025-02-15 10:05:18,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:05:18,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:05:18,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:05:18,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-15 10:05:18,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24483.44 MB 2025-02-15 10:05:18,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:05:18,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 10:05:18,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29062.33 MB 2025-02-15 10:05:18,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:18,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25900.87 MB 2025-02-15 10:05:18,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:05:18,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:05:18,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:05:18,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24483.44 MB 2025-02-15 10:05:18,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-15 10:05:18,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:05:18,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 10:05:18,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 10:05:18,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:05:18,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-15 10:05:18,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:05:18,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:05:18,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:05:18,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-15 10:05:18,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-15 10:05:18,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:05:18,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 10:05:18,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 10:05:18,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:05:18,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-15 10:05:18,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:05:18,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:05:18,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:05:18,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28258.84 MB 2025-02-15 10:05:18,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29025.84 MB 2025-02-15 10:05:18,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:05:18,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 10:05:18,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35139.88 MB 2025-02-15 10:05:18,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:05:18,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29733.63 MB 2025-02-15 10:05:18,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:05:18,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:05:18,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:18,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29438.73 MB 2025-02-15 10:05:18,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29667.52 MB 2025-02-15 10:05:18,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 10:05:18,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35139.88 MB 2025-02-15 10:05:18,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35139.88 MB 2025-02-15 10:05:18,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:18,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29895.46 MB 2025-02-15 10:05:18,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:05:18,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:05:18,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.24 seconds 2025-02-15 10:05:18,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:18,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17180.96 MB 2025-02-15 10:05:18,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29868.37 MB 2025-02-15 10:05:18,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12687.41 MB 2025-02-15 10:05:18,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55171.87 MB 2025-02-15 10:05:18,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35139.88 MB 2025-02-15 10:05:18,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20032.00 MB 2025-02-15 10:05:18,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29895.46 MB 2025-02-15 10:05:19,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:05:19,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:05:19,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:05:19,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:19,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29868.37 MB 2025-02-15 10:05:19,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22179.78 MB 2025-02-15 10:05:19,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7688.59 MB 2025-02-15 10:05:19,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35139.88 MB 2025-02-15 10:05:19,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35139.88 MB 2025-02-15 10:05:19,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:19,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32375.43 MB 2025-02-15 10:05:19,058 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 10:05:19,058 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:05:19,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:05:19,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:05:19,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:19,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:19,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22179.78 MB 2025-02-15 10:05:19,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30602.99 MB 2025-02-15 10:05:19,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 10:05:19,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35139.88 MB 2025-02-15 10:05:19,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43515.90 MB 2025-02-15 10:05:19,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 10:05:19,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30602.99 MB 2025-02-15 10:05:19,225 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 10:05:19,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:19,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:05:19,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:19,228 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:05:19,233 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:05:19,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:19,235 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:05:19,235 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:05:30,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:30,467 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:05:30,472 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:05:30,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:30,475 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 519, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:05:30,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:30,476 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 519, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:05:38,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:05:38,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:05:38,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.12 seconds 2025-02-15 10:05:38,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:38,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16585.18 MB 2025-02-15 10:05:38,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18421.89 MB 2025-02-15 10:05:38,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1836.71 MB 2025-02-15 10:05:38,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51891.93 MB 2025-02-15 10:05:38,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20254.29 MB 2025-02-15 10:05:38,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31637.64 MB 2025-02-15 10:05:38,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27415.51 MB 2025-02-15 10:05:38,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:05:38,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:05:38,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 10:05:38,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:38,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18421.89 MB 2025-02-15 10:05:38,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18477.02 MB 2025-02-15 10:05:38,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 55.12 MB 2025-02-15 10:05:38,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20254.29 MB 2025-02-15 10:05:38,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25717.37 MB 2025-02-15 10:05:38,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5463.08 MB 2025-02-15 10:05:38,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25985.04 MB 2025-02-15 10:05:40,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:05:40,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:05:40,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:05:40,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:40,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18477.02 MB 2025-02-15 10:05:40,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19007.86 MB 2025-02-15 10:05:40,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:05:40,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25717.37 MB 2025-02-15 10:05:40,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21434.99 MB 2025-02-15 10:05:40,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4282.38 MB 2025-02-15 10:05:40,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22988.48 MB 2025-02-15 10:05:40,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:05:40,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:05:40,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:05:40,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:40,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19007.86 MB 2025-02-15 10:05:40,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20897.39 MB 2025-02-15 10:05:40,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:05:40,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 10:05:40,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24266.15 MB 2025-02-15 10:05:40,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:05:40,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22314.82 MB 2025-02-15 10:05:40,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:05:40,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:05:40,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:05:40,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:40,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20897.39 MB 2025-02-15 10:05:40,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23139.25 MB 2025-02-15 10:05:40,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:05:40,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24266.15 MB 2025-02-15 10:05:40,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 10:05:40,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:05:40,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28683.53 MB 2025-02-15 10:05:40,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:05:40,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:05:40,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:05:40,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:40,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19007.86 MB 2025-02-15 10:05:40,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23139.25 MB 2025-02-15 10:05:40,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:05:40,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21434.99 MB 2025-02-15 10:05:40,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-15 10:05:40,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:05:40,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28683.53 MB 2025-02-15 10:05:40,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:05:40,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:05:40,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:05:40,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:40,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24672.79 MB 2025-02-15 10:05:40,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25439.79 MB 2025-02-15 10:05:40,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:05:40,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-15 10:05:40,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:05:40,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:05:40,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26147.58 MB 2025-02-15 10:05:41,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:05:41,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:05:41,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:41,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:41,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25852.68 MB 2025-02-15 10:05:41,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26082.65 MB 2025-02-15 10:05:41,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.97 MB 2025-02-15 10:05:41,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:05:41,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:05:41,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:41,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26241.33 MB 2025-02-15 10:05:41,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:05:41,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:05:41,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.53 seconds 2025-02-15 10:05:41,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:41,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14776.94 MB 2025-02-15 10:05:41,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26283.72 MB 2025-02-15 10:05:41,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11506.78 MB 2025-02-15 10:05:41,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51891.93 MB 2025-02-15 10:05:41,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:05:41,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21076.38 MB 2025-02-15 10:05:41,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26283.72 MB 2025-02-15 10:05:41,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:05:41,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:05:41,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:05:41,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:41,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26283.72 MB 2025-02-15 10:05:41,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19781.33 MB 2025-02-15 10:05:41,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.39 MB 2025-02-15 10:05:41,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:05:41,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 10:05:41,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:41,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28795.39 MB 2025-02-15 10:05:41,297 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:05:41,297 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:05:41,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:05:41,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:05:41,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:41,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:41,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19781.33 MB 2025-02-15 10:05:41,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28220.36 MB 2025-02-15 10:05:41,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:05:41,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 10:05:41,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 10:05:41,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:05:41,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28220.36 MB 2025-02-15 10:05:41,467 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:05:41,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:41,468 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:05:41,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:41,469 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:05:41,474 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:05:41,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:41,475 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:05:41,475 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:05:51,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:51,857 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:05:51,864 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:05:51,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:51,870 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:05:51,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:51,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:05:55,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:05:55,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:05:55,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.58 seconds 2025-02-15 10:05:55,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:55,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14522.61 MB 2025-02-15 10:05:55,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.79 MB 2025-02-15 10:05:55,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-15 10:05:55,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 10:05:55,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18605.93 MB 2025-02-15 10:05:55,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35284.58 MB 2025-02-15 10:05:55,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24220.47 MB 2025-02-15 10:05:55,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:05:55,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:05:55,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:55,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:55,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.79 MB 2025-02-15 10:05:55,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15596.42 MB 2025-02-15 10:05:55,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.63 MB 2025-02-15 10:05:55,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18605.93 MB 2025-02-15 10:05:55,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20086.52 MB 2025-02-15 10:05:55,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1480.59 MB 2025-02-15 10:05:55,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18248.72 MB 2025-02-15 10:05:56,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:05:56,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:05:56,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.05 seconds 2025-02-15 10:05:56,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15596.42 MB 2025-02-15 10:05:56,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15873.78 MB 2025-02-15 10:05:56,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-15 10:05:56,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20086.52 MB 2025-02-15 10:05:56,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19346.23 MB 2025-02-15 10:05:56,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -740.29 MB 2025-02-15 10:05:56,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19852.04 MB 2025-02-15 10:05:56,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:05:56,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:05:56,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:05:56,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.78 MB 2025-02-15 10:05:56,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16860.82 MB 2025-02-15 10:05:56,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-15 10:05:56,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19346.23 MB 2025-02-15 10:05:56,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19841.16 MB 2025-02-15 10:05:56,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 494.93 MB 2025-02-15 10:05:56,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17601.43 MB 2025-02-15 10:05:56,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:05:56,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:05:56,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:05:56,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16860.82 MB 2025-02-15 10:05:56,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18032.22 MB 2025-02-15 10:05:56,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.40 MB 2025-02-15 10:05:56,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19841.16 MB 2025-02-15 10:05:56,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22810.72 MB 2025-02-15 10:05:56,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-15 10:05:56,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20930.92 MB 2025-02-15 10:05:56,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:05:56,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:05:56,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:05:56,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.78 MB 2025-02-15 10:05:56,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18032.22 MB 2025-02-15 10:05:56,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.44 MB 2025-02-15 10:05:56,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19346.23 MB 2025-02-15 10:05:56,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22810.72 MB 2025-02-15 10:05:56,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3464.50 MB 2025-02-15 10:05:56,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20930.92 MB 2025-02-15 10:05:56,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:05:56,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:05:56,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 10:05:56,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18833.50 MB 2025-02-15 10:05:56,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19236.09 MB 2025-02-15 10:05:56,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.59 MB 2025-02-15 10:05:56,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22810.72 MB 2025-02-15 10:05:56,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:05:56,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 10:05:56,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19608.61 MB 2025-02-15 10:05:56,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:05:56,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:05:56,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:05:56,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19451.83 MB 2025-02-15 10:05:56,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19680.45 MB 2025-02-15 10:05:56,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.61 MB 2025-02-15 10:05:56,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23024.63 MB 2025-02-15 10:05:56,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:05:56,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:56,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19736.55 MB 2025-02-15 10:05:56,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:05:56,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:05:56,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.00 seconds 2025-02-15 10:05:56,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:56,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13745.66 MB 2025-02-15 10:05:56,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19881.52 MB 2025-02-15 10:05:56,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6135.86 MB 2025-02-15 10:05:56,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 10:05:56,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:05:56,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30865.88 MB 2025-02-15 10:05:56,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19881.52 MB 2025-02-15 10:05:57,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:05:57,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:05:57,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 10:05:57,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:57,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14836.20 MB 2025-02-15 10:05:57,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17850.23 MB 2025-02-15 10:05:57,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:05:57,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23024.63 MB 2025-02-15 10:05:57,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:05:57,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:05:57,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18151.60 MB 2025-02-15 10:05:57,179 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:05:57,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:05:57,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:05:57,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:05:57,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:05:57,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:05:57,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17850.23 MB 2025-02-15 10:05:57,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26289.26 MB 2025-02-15 10:05:57,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:05:57,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23024.63 MB 2025-02-15 10:05:57,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31415.34 MB 2025-02-15 10:05:57,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:05:57,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26289.26 MB 2025-02-15 10:05:57,438 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:05:57,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:57,441 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:05:57,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:57,443 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:05:57,450 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:05:57,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:05:57,452 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:05:57,453 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:06:26,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:26,936 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:06:26,941 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:06:26,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:26,946 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 194, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:06:26,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:26,947 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 194, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:06:29,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:06:29,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:06:29,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-15 10:06:29,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:29,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14320.53 MB 2025-02-15 10:06:29,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15007.08 MB 2025-02-15 10:06:29,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 686.56 MB 2025-02-15 10:06:29,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44000.35 MB 2025-02-15 10:06:29,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 10:06:29,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23928.50 MB 2025-02-15 10:06:29,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24018.39 MB 2025-02-15 10:06:30,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:06:30,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:06:30,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:06:30,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:30,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15007.08 MB 2025-02-15 10:06:30,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.70 MB 2025-02-15 10:06:30,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.61 MB 2025-02-15 10:06:30,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 10:06:30,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 10:06:30,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:06:30,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17718.04 MB 2025-02-15 10:06:30,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:06:30,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:06:30,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 10:06:30,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:30,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.70 MB 2025-02-15 10:06:30,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15588.83 MB 2025-02-15 10:06:30,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 10:06:30,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 10:06:30,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19599.98 MB 2025-02-15 10:06:30,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 10:06:30,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19588.32 MB 2025-02-15 10:06:30,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:06:30,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:06:30,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:06:30,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:30,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15588.76 MB 2025-02-15 10:06:30,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16500.24 MB 2025-02-15 10:06:30,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 10:06:30,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19599.98 MB 2025-02-15 10:06:30,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19599.98 MB 2025-02-15 10:06:30,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:06:30,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17184.15 MB 2025-02-15 10:06:31,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:06:31,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:06:31,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:06:31,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16500.24 MB 2025-02-15 10:06:31,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17581.97 MB 2025-02-15 10:06:31,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 10:06:31,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19599.98 MB 2025-02-15 10:06:31,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21885.88 MB 2025-02-15 10:06:31,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 10:06:31,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20257.05 MB 2025-02-15 10:06:31,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:06:31,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:06:31,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:06:31,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15588.76 MB 2025-02-15 10:06:31,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17581.97 MB 2025-02-15 10:06:31,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 10:06:31,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19599.98 MB 2025-02-15 10:06:31,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21885.88 MB 2025-02-15 10:06:31,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 10:06:31,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20257.05 MB 2025-02-15 10:06:31,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:06:31,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:06:31,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:06:31,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18321.90 MB 2025-02-15 10:06:31,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18691.98 MB 2025-02-15 10:06:31,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 10:06:31,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21885.88 MB 2025-02-15 10:06:31,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 10:06:31,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-15 10:06:31,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19036.15 MB 2025-02-15 10:06:31,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:06:31,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:06:31,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:06:31,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18891.21 MB 2025-02-15 10:06:31,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19120.23 MB 2025-02-15 10:06:31,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-15 10:06:31,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22085.11 MB 2025-02-15 10:06:31,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 10:06:31,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:06:31,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19158.87 MB 2025-02-15 10:06:31,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:06:31,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:06:31,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.20 seconds 2025-02-15 10:06:31,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13644.62 MB 2025-02-15 10:06:31,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.30 MB 2025-02-15 10:06:31,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.68 MB 2025-02-15 10:06:31,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44000.35 MB 2025-02-15 10:06:31,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 10:06:31,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21913.14 MB 2025-02-15 10:06:31,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.30 MB 2025-02-15 10:06:31,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:06:31,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:06:31,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:06:31,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.30 MB 2025-02-15 10:06:31,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17672.11 MB 2025-02-15 10:06:31,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1649.19 MB 2025-02-15 10:06:31,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 10:06:31,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22087.20 MB 2025-02-15 10:06:31,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:06:31,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.30 MB 2025-02-15 10:06:31,438 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:06:31,439 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:06:31,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:06:31,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:06:31,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:06:31,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:06:31,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.11 MB 2025-02-15 10:06:31,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26112.09 MB 2025-02-15 10:06:31,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.97 MB 2025-02-15 10:06:31,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22087.20 MB 2025-02-15 10:06:31,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32577.16 MB 2025-02-15 10:06:31,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:06:31,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26112.09 MB 2025-02-15 10:06:31,607 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:06:31,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:31,608 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:06:31,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:31,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:06:31,614 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:06:31,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:06:31,615 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:06:31,615 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:07:28,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:28,658 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:07:28,663 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:07:28,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:28,667 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 608, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:07:28,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:28,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 608, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:07:38,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:07:38,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:07:38,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.38 seconds 2025-02-15 10:07:38,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:38,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17205.35 MB 2025-02-15 10:07:38,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19357.03 MB 2025-02-15 10:07:38,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2151.68 MB 2025-02-15 10:07:38,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45162.17 MB 2025-02-15 10:07:38,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22366.13 MB 2025-02-15 10:07:38,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22796.04 MB 2025-02-15 10:07:38,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28262.97 MB 2025-02-15 10:07:38,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:07:38,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:07:38,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 10:07:38,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:38,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19357.03 MB 2025-02-15 10:07:38,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18609.23 MB 2025-02-15 10:07:38,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -747.80 MB 2025-02-15 10:07:38,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22366.13 MB 2025-02-15 10:07:38,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26096.96 MB 2025-02-15 10:07:38,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3730.83 MB 2025-02-15 10:07:38,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24317.74 MB 2025-02-15 10:07:39,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:07:39,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:07:39,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-15 10:07:39,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:39,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18609.23 MB 2025-02-15 10:07:39,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19077.69 MB 2025-02-15 10:07:39,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 468.47 MB 2025-02-15 10:07:39,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26096.96 MB 2025-02-15 10:07:39,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21464.35 MB 2025-02-15 10:07:39,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4632.61 MB 2025-02-15 10:07:39,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23034.72 MB 2025-02-15 10:07:39,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:07:39,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:07:39,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:07:39,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:39,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.69 MB 2025-02-15 10:07:39,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20745.19 MB 2025-02-15 10:07:39,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1667.50 MB 2025-02-15 10:07:39,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21464.35 MB 2025-02-15 10:07:39,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23968.35 MB 2025-02-15 10:07:39,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2504.00 MB 2025-02-15 10:07:39,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21996.07 MB 2025-02-15 10:07:39,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:07:39,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:07:39,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:07:39,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:39,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20745.19 MB 2025-02-15 10:07:39,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22723.64 MB 2025-02-15 10:07:39,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.45 MB 2025-02-15 10:07:39,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23968.35 MB 2025-02-15 10:07:39,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29393.68 MB 2025-02-15 10:07:39,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5425.33 MB 2025-02-15 10:07:39,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27619.21 MB 2025-02-15 10:07:39,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:07:39,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:07:39,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 10:07:39,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:39,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.69 MB 2025-02-15 10:07:39,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22723.64 MB 2025-02-15 10:07:39,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3645.94 MB 2025-02-15 10:07:39,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21464.35 MB 2025-02-15 10:07:39,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29393.68 MB 2025-02-15 10:07:39,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7929.33 MB 2025-02-15 10:07:39,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27619.21 MB 2025-02-15 10:07:40,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:07:40,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:07:40,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:07:40,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:40,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24076.99 MB 2025-02-15 10:07:40,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24755.70 MB 2025-02-15 10:07:40,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 678.71 MB 2025-02-15 10:07:40,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29393.68 MB 2025-02-15 10:07:40,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29760.68 MB 2025-02-15 10:07:40,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 367.00 MB 2025-02-15 10:07:40,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25380.33 MB 2025-02-15 10:07:40,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:07:40,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:07:40,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:07:40,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:40,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25120.08 MB 2025-02-15 10:07:40,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25356.44 MB 2025-02-15 10:07:40,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.36 MB 2025-02-15 10:07:40,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29760.68 MB 2025-02-15 10:07:40,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29762.78 MB 2025-02-15 10:07:40,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 10:07:40,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25484.57 MB 2025-02-15 10:07:40,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:07:40,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:07:40,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.48 seconds 2025-02-15 10:07:40,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:40,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15087.03 MB 2025-02-15 10:07:40,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25557.51 MB 2025-02-15 10:07:40,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10470.48 MB 2025-02-15 10:07:40,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45162.17 MB 2025-02-15 10:07:40,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29762.78 MB 2025-02-15 10:07:40,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15399.39 MB 2025-02-15 10:07:40,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25557.51 MB 2025-02-15 10:07:40,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:07:40,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:07:40,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:07:40,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:40,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25557.51 MB 2025-02-15 10:07:40,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28571.54 MB 2025-02-15 10:07:40,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:07:40,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29762.78 MB 2025-02-15 10:07:40,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29762.78 MB 2025-02-15 10:07:40,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:07:40,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28872.91 MB 2025-02-15 10:07:40,441 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:07:40,441 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:07:40,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:07:40,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:07:40,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:07:40,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:07:40,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19871.18 MB 2025-02-15 10:07:40,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28310.21 MB 2025-02-15 10:07:40,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:07:40,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29762.78 MB 2025-02-15 10:07:40,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40252.74 MB 2025-02-15 10:07:40,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:07:40,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28310.21 MB 2025-02-15 10:07:40,605 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:07:40,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:40,606 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:07:40,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:40,607 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:07:40,612 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:07:40,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:40,613 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:07:40,613 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:07:49,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:49,206 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:07:49,211 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:07:49,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:49,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1150, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:07:49,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:07:49,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1150, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:08:07,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:08:07,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:08:07,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.90 seconds 2025-02-15 10:08:07,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:07,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.09 MB 2025-02-15 10:08:07,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25052.66 MB 2025-02-15 10:08:07,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4070.57 MB 2025-02-15 10:08:07,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52837.74 MB 2025-02-15 10:08:07,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28940.70 MB 2025-02-15 10:08:07,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23897.05 MB 2025-02-15 10:08:07,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33850.85 MB 2025-02-15 10:08:07,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:08:07,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:08:07,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:08:07,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:07,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25052.66 MB 2025-02-15 10:08:07,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21757.39 MB 2025-02-15 10:08:07,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3295.27 MB 2025-02-15 10:08:07,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28940.70 MB 2025-02-15 10:08:07,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39095.11 MB 2025-02-15 10:08:07,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10154.41 MB 2025-02-15 10:08:07,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37307.25 MB 2025-02-15 10:08:09,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:08:09,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:08:09,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 10:08:09,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21757.39 MB 2025-02-15 10:08:09,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22288.23 MB 2025-02-15 10:08:09,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:08:09,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39095.11 MB 2025-02-15 10:08:09,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24895.29 MB 2025-02-15 10:08:09,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14199.82 MB 2025-02-15 10:08:09,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26267.82 MB 2025-02-15 10:08:09,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:08:09,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:08:09,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:08:09,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-15 10:08:09,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24177.77 MB 2025-02-15 10:08:09,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:08:09,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 10:08:09,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26782.73 MB 2025-02-15 10:08:09,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:08:09,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25595.20 MB 2025-02-15 10:08:09,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:08:09,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:08:09,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:08:09,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24177.77 MB 2025-02-15 10:08:09,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.62 MB 2025-02-15 10:08:09,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:08:09,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26782.73 MB 2025-02-15 10:08:09,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34097.59 MB 2025-02-15 10:08:09,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 10:08:09,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.90 MB 2025-02-15 10:08:09,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:08:09,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:08:09,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:08:09,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-15 10:08:09,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.62 MB 2025-02-15 10:08:09,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:08:09,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 10:08:09,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34097.59 MB 2025-02-15 10:08:09,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-15 10:08:09,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.90 MB 2025-02-15 10:08:09,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:08:09,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:08:09,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:08:09,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27953.16 MB 2025-02-15 10:08:09,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28720.17 MB 2025-02-15 10:08:09,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:08:09,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34097.59 MB 2025-02-15 10:08:09,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34514.93 MB 2025-02-15 10:08:09,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:08:09,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29427.96 MB 2025-02-15 10:08:09,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:08:09,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:08:09,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:08:09,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29133.06 MB 2025-02-15 10:08:09,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.50 MB 2025-02-15 10:08:09,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 10:08:09,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34514.93 MB 2025-02-15 10:08:09,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34514.93 MB 2025-02-15 10:08:09,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:09,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29597.09 MB 2025-02-15 10:08:09,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:08:09,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:08:09,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.40 seconds 2025-02-15 10:08:09,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16975.40 MB 2025-02-15 10:08:09,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29562.35 MB 2025-02-15 10:08:09,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12586.95 MB 2025-02-15 10:08:09,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52837.74 MB 2025-02-15 10:08:09,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34514.93 MB 2025-02-15 10:08:09,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18322.82 MB 2025-02-15 10:08:09,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29597.09 MB 2025-02-15 10:08:09,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:08:09,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:08:09,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:08:09,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29562.35 MB 2025-02-15 10:08:09,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21969.23 MB 2025-02-15 10:08:09,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7593.12 MB 2025-02-15 10:08:09,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34514.93 MB 2025-02-15 10:08:09,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34514.93 MB 2025-02-15 10:08:09,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:09,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32065.11 MB 2025-02-15 10:08:09,902 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 10:08:09,902 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:08:09,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:08:09,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:08:09,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:08:09,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:09,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21969.23 MB 2025-02-15 10:08:09,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30378.54 MB 2025-02-15 10:08:09,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 10:08:09,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34514.93 MB 2025-02-15 10:08:09,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42874.18 MB 2025-02-15 10:08:09,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 10:08:09,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30378.54 MB 2025-02-15 10:08:10,072 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 10:08:10,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:10,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:08:10,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:10,075 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:08:10,079 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:08:10,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:10,081 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:08:10,081 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:08:45,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:45,708 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:08:45,713 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:08:45,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:45,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:08:45,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:45,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:08:48,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:08:48,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:08:48,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-15 10:08:48,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:48,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14118.45 MB 2025-02-15 10:08:48,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.38 MB 2025-02-15 10:08:48,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-15 10:08:48,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51233.42 MB 2025-02-15 10:08:48,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18836.62 MB 2025-02-15 10:08:48,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32396.80 MB 2025-02-15 10:08:48,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23589.82 MB 2025-02-15 10:08:48,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:08:48,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:08:48,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:08:48,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:48,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.38 MB 2025-02-15 10:08:48,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14858.88 MB 2025-02-15 10:08:48,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 156.50 MB 2025-02-15 10:08:48,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18836.62 MB 2025-02-15 10:08:48,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18836.62 MB 2025-02-15 10:08:48,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:48,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16767.21 MB 2025-02-15 10:08:49,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:08:49,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:08:49,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 10:08:49,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14858.88 MB 2025-02-15 10:08:49,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15053.96 MB 2025-02-15 10:08:49,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-15 10:08:49,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18836.62 MB 2025-02-15 10:08:49,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 10:08:49,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 10:08:49,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.56 MB 2025-02-15 10:08:49,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:08:49,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:08:49,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:08:49,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15053.89 MB 2025-02-15 10:08:49,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15748.13 MB 2025-02-15 10:08:49,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-15 10:08:49,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 10:08:49,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18364.76 MB 2025-02-15 10:08:49,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:49,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16269.04 MB 2025-02-15 10:08:49,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:08:49,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:08:49,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:08:49,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15748.13 MB 2025-02-15 10:08:49,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16572.05 MB 2025-02-15 10:08:49,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-15 10:08:49,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 10:08:49,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 10:08:49,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 10:08:49,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18609.54 MB 2025-02-15 10:08:49,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:08:49,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:08:49,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:08:49,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15053.89 MB 2025-02-15 10:08:49,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16572.05 MB 2025-02-15 10:08:49,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-15 10:08:49,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18364.76 MB 2025-02-15 10:08:49,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 10:08:49,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 10:08:49,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18609.54 MB 2025-02-15 10:08:49,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:08:49,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:08:49,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 10:08:49,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17135.63 MB 2025-02-15 10:08:49,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17417.50 MB 2025-02-15 10:08:49,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-15 10:08:49,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 10:08:49,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19908.26 MB 2025-02-15 10:08:49,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 10:08:49,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17689.08 MB 2025-02-15 10:08:49,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:08:49,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:08:49,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:08:49,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17569.25 MB 2025-02-15 10:08:49,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17777.32 MB 2025-02-15 10:08:49,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.07 MB 2025-02-15 10:08:49,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19908.26 MB 2025-02-15 10:08:49,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19908.26 MB 2025-02-15 10:08:49,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:49,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17787.79 MB 2025-02-15 10:08:49,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:08:49,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:08:49,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-15 10:08:49,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13543.58 MB 2025-02-15 10:08:49,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17977.97 MB 2025-02-15 10:08:49,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4434.39 MB 2025-02-15 10:08:49,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51233.42 MB 2025-02-15 10:08:49,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19908.26 MB 2025-02-15 10:08:49,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31325.16 MB 2025-02-15 10:08:49,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17977.97 MB 2025-02-15 10:08:49,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:08:49,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:08:49,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:08:49,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17977.97 MB 2025-02-15 10:08:49,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17347.09 MB 2025-02-15 10:08:49,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -630.88 MB 2025-02-15 10:08:49,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19908.26 MB 2025-02-15 10:08:49,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19908.26 MB 2025-02-15 10:08:49,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:08:49,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19080.81 MB 2025-02-15 10:08:49,473 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 10:08:49,473 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 10:08:49,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:08:49,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:08:49,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:08:49,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:08:49,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17347.09 MB 2025-02-15 10:08:49,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25769.05 MB 2025-02-15 10:08:49,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-15 10:08:49,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19908.26 MB 2025-02-15 10:08:49,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30373.05 MB 2025-02-15 10:08:49,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 10:08:49,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25769.05 MB 2025-02-15 10:08:49,644 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 10:08:49,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:49,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:08:49,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:49,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:08:49,651 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:08:49,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:08:49,652 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:08:49,652 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 10:09:10,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:10,098 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:09:10,103 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:09:10,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:10,106 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:09:10,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:10,107 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:09:22,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:09:22,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:09:22,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.61 seconds 2025-02-15 10:09:22,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:22,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-15 10:09:22,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-15 10:09:22,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-15 10:09:22,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38744.88 MB 2025-02-15 10:09:22,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25616.71 MB 2025-02-15 10:09:22,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13128.17 MB 2025-02-15 10:09:22,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.08 MB 2025-02-15 10:09:22,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:09:22,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:09:22,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:09:22,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:22,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-15 10:09:22,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19978.39 MB 2025-02-15 10:09:22,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1481.10 MB 2025-02-15 10:09:22,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25616.71 MB 2025-02-15 10:09:22,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-15 10:09:22,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7050.63 MB 2025-02-15 10:09:22,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31318.02 MB 2025-02-15 10:09:24,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:09:24,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:09:24,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:09:24,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:24,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.39 MB 2025-02-15 10:09:24,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20509.23 MB 2025-02-15 10:09:24,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:09:24,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32667.34 MB 2025-02-15 10:09:24,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24171.77 MB 2025-02-15 10:09:24,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8495.56 MB 2025-02-15 10:09:24,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24487.78 MB 2025-02-15 10:09:24,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:09:24,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:09:24,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:09:24,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:24,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-15 10:09:24,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22398.77 MB 2025-02-15 10:09:24,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:09:24,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24171.77 MB 2025-02-15 10:09:24,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26059.21 MB 2025-02-15 10:09:24,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:09:24,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23816.20 MB 2025-02-15 10:09:24,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:09:24,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:09:24,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:09:24,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:24,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.77 MB 2025-02-15 10:09:24,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-15 10:09:24,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:09:24,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26059.21 MB 2025-02-15 10:09:24,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32665.24 MB 2025-02-15 10:09:24,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:09:24,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-15 10:09:24,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:09:24,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:09:24,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:09:24,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:24,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-15 10:09:24,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-15 10:09:24,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:09:24,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24171.77 MB 2025-02-15 10:09:24,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32665.24 MB 2025-02-15 10:09:24,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 10:09:24,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-15 10:09:25,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:09:25,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:09:25,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:09:25,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:25,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26174.16 MB 2025-02-15 10:09:25,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26941.17 MB 2025-02-15 10:09:25,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:09:25,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32665.24 MB 2025-02-15 10:09:25,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-15 10:09:25,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:09:25,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27648.95 MB 2025-02-15 10:09:25,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:09:25,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:09:25,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:09:25,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:25,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.06 MB 2025-02-15 10:09:25,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27582.50 MB 2025-02-15 10:09:25,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-15 10:09:25,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-15 10:09:25,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-15 10:09:25,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:09:25,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27800.13 MB 2025-02-15 10:09:25,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:09:25,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:09:25,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.03 seconds 2025-02-15 10:09:25,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:25,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-15 10:09:25,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27782.86 MB 2025-02-15 10:09:25,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11999.02 MB 2025-02-15 10:09:25,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38744.88 MB 2025-02-15 10:09:25,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-15 10:09:25,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5662.31 MB 2025-02-15 10:09:25,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27800.13 MB 2025-02-15 10:09:25,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:09:25,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:09:25,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:09:25,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:25,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27782.86 MB 2025-02-15 10:09:25,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20777.68 MB 2025-02-15 10:09:25,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7005.18 MB 2025-02-15 10:09:25,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-15 10:09:25,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-15 10:09:25,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:09:25,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30285.62 MB 2025-02-15 10:09:25,426 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 10:09:25,426 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:09:25,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:09:25,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:09:25,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:09:25,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:09:25,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20777.68 MB 2025-02-15 10:09:25,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29187.07 MB 2025-02-15 10:09:25,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.40 MB 2025-02-15 10:09:25,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-15 10:09:25,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41441.82 MB 2025-02-15 10:09:25,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 10:09:25,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29187.07 MB 2025-02-15 10:09:25,596 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 10:09:25,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:25,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:09:25,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:25,599 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:09:25,603 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:09:25,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:09:25,604 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:09:25,605 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:10:16,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:16,457 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:10:16,462 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:10:16,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:16,466 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 406, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:10:16,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:16,467 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 406, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:10:22,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:10:22,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:10:22,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.28 seconds 2025-02-15 10:10:22,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:22,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15797.78 MB 2025-02-15 10:10:22,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17234.59 MB 2025-02-15 10:10:22,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1436.81 MB 2025-02-15 10:10:22,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49801.07 MB 2025-02-15 10:10:22,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19308.48 MB 2025-02-15 10:10:22,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30492.59 MB 2025-02-15 10:10:22,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26175.12 MB 2025-02-15 10:10:22,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:10:22,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:10:22,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:10:22,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:22,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17234.59 MB 2025-02-15 10:10:22,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17791.06 MB 2025-02-15 10:10:22,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 556.47 MB 2025-02-15 10:10:22,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19308.48 MB 2025-02-15 10:10:22,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24052.24 MB 2025-02-15 10:10:22,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4743.76 MB 2025-02-15 10:10:22,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22658.74 MB 2025-02-15 10:10:24,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:10:24,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:10:24,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.86 seconds 2025-02-15 10:10:24,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:24,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17791.06 MB 2025-02-15 10:10:24,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18303.32 MB 2025-02-15 10:10:24,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-15 10:10:24,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24052.24 MB 2025-02-15 10:10:24,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20394.80 MB 2025-02-15 10:10:24,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3657.43 MB 2025-02-15 10:10:24,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22301.48 MB 2025-02-15 10:10:24,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:10:24,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:10:24,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:10:24,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:24,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.32 MB 2025-02-15 10:10:24,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20126.56 MB 2025-02-15 10:10:24,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.24 MB 2025-02-15 10:10:24,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20394.80 MB 2025-02-15 10:10:24,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23131.59 MB 2025-02-15 10:10:24,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2736.78 MB 2025-02-15 10:10:24,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21494.38 MB 2025-02-15 10:10:24,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:10:24,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:10:24,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:10:24,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:24,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20126.56 MB 2025-02-15 10:10:24,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22290.87 MB 2025-02-15 10:10:24,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2164.31 MB 2025-02-15 10:10:24,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23131.59 MB 2025-02-15 10:10:24,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29521.61 MB 2025-02-15 10:10:24,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6390.02 MB 2025-02-15 10:10:24,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27641.10 MB 2025-02-15 10:10:24,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:10:24,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:10:24,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:10:24,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:24,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.32 MB 2025-02-15 10:10:24,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22290.87 MB 2025-02-15 10:10:24,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3987.55 MB 2025-02-15 10:10:24,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20394.80 MB 2025-02-15 10:10:24,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29521.61 MB 2025-02-15 10:10:24,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9126.81 MB 2025-02-15 10:10:24,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27641.10 MB 2025-02-15 10:10:25,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:10:25,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:10:25,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 10:10:25,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:25,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23770.74 MB 2025-02-15 10:10:25,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24510.90 MB 2025-02-15 10:10:25,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-15 10:10:25,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29521.61 MB 2025-02-15 10:10:25,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29924.26 MB 2025-02-15 10:10:25,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 10:10:25,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25193.91 MB 2025-02-15 10:10:25,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:10:25,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:10:25,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:10:25,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:25,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24909.34 MB 2025-02-15 10:10:25,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25115.79 MB 2025-02-15 10:10:25,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.46 MB 2025-02-15 10:10:25,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29924.26 MB 2025-02-15 10:10:25,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29928.46 MB 2025-02-15 10:10:25,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 10:10:25,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25303.08 MB 2025-02-15 10:10:25,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:10:25,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:10:25,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.62 seconds 2025-02-15 10:10:25,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:25,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14383.24 MB 2025-02-15 10:10:25,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25316.87 MB 2025-02-15 10:10:25,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10933.62 MB 2025-02-15 10:10:25,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49801.07 MB 2025-02-15 10:10:25,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29928.46 MB 2025-02-15 10:10:25,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19872.61 MB 2025-02-15 10:10:25,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25316.87 MB 2025-02-15 10:10:25,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:10:25,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:10:25,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:10:25,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:25,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25316.87 MB 2025-02-15 10:10:25,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.04 MB 2025-02-15 10:10:25,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5995.83 MB 2025-02-15 10:10:25,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29928.46 MB 2025-02-15 10:10:25,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29928.46 MB 2025-02-15 10:10:25,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:10:25,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28029.47 MB 2025-02-15 10:10:25,377 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:10:25,378 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:10:25,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:10:25,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:10:25,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:10:25,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:10:25,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.04 MB 2025-02-15 10:10:25,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27760.06 MB 2025-02-15 10:10:25,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:10:25,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29928.46 MB 2025-02-15 10:10:25,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38319.16 MB 2025-02-15 10:10:25,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:10:25,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27760.06 MB 2025-02-15 10:10:25,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:10:25,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:25,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:10:25,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:25,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:10:25,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:10:25,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:10:25,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:10:25,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:11:23,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:23,844 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:11:23,849 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:11:23,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:23,853 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:11:23,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:23,854 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:11:42,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:11:42,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:11:42,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.78 seconds 2025-02-15 10:11:42,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:42,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-15 10:11:42,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-15 10:11:42,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 10:11:42,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50904.17 MB 2025-02-15 10:11:42,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-15 10:11:42,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17536.39 MB 2025-02-15 10:11:42,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-15 10:11:42,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:11:42,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:11:42,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:11:42,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:42,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-15 10:11:42,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20966.92 MB 2025-02-15 10:11:42,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4789.58 MB 2025-02-15 10:11:42,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33367.79 MB 2025-02-15 10:11:42,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-15 10:11:42,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:11:42,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29099.46 MB 2025-02-15 10:11:43,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:11:43,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:11:43,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.14 seconds 2025-02-15 10:11:43,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:43,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20966.92 MB 2025-02-15 10:11:43,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21282.77 MB 2025-02-15 10:11:43,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.85 MB 2025-02-15 10:11:43,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33367.79 MB 2025-02-15 10:11:43,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 10:11:43,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4307.55 MB 2025-02-15 10:11:43,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25221.51 MB 2025-02-15 10:11:43,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:11:43,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:11:43,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:11:43,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:43,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21282.77 MB 2025-02-15 10:11:43,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22406.77 MB 2025-02-15 10:11:43,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1124.00 MB 2025-02-15 10:11:43,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 10:11:43,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 10:11:43,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:11:43,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23250.15 MB 2025-02-15 10:11:43,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:11:43,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:11:43,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:11:43,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:43,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22406.77 MB 2025-02-15 10:11:43,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23740.70 MB 2025-02-15 10:11:43,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1333.93 MB 2025-02-15 10:11:43,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 10:11:43,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 10:11:43,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:11:43,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27039.53 MB 2025-02-15 10:11:43,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:11:43,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:11:43,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:11:43,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:43,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21282.77 MB 2025-02-15 10:11:43,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23740.70 MB 2025-02-15 10:11:43,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2457.93 MB 2025-02-15 10:11:43,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 10:11:43,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 10:11:43,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:11:43,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27039.53 MB 2025-02-15 10:11:44,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:11:44,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:11:44,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:11:44,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:44,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24653.16 MB 2025-02-15 10:11:44,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25109.53 MB 2025-02-15 10:11:44,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 456.37 MB 2025-02-15 10:11:44,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 10:11:44,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29307.70 MB 2025-02-15 10:11:44,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 247.46 MB 2025-02-15 10:11:44,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25530.66 MB 2025-02-15 10:11:44,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:11:44,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:11:44,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:11:44,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:44,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25355.20 MB 2025-02-15 10:11:44,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25576.34 MB 2025-02-15 10:11:44,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.14 MB 2025-02-15 10:11:44,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29307.70 MB 2025-02-15 10:11:44,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29309.80 MB 2025-02-15 10:11:44,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 10:11:44,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25633.75 MB 2025-02-15 10:11:44,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:11:44,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:11:44,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.23 seconds 2025-02-15 10:11:44,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:44,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-15 10:11:44,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25777.42 MB 2025-02-15 10:11:44,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8568.58 MB 2025-02-15 10:11:44,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50904.17 MB 2025-02-15 10:11:44,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29309.80 MB 2025-02-15 10:11:44,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21594.37 MB 2025-02-15 10:11:44,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25777.42 MB 2025-02-15 10:11:44,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:11:44,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:11:44,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:11:44,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:44,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25777.42 MB 2025-02-15 10:11:44,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28791.45 MB 2025-02-15 10:11:44,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:11:44,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29309.80 MB 2025-02-15 10:11:44,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29980.88 MB 2025-02-15 10:11:44,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 671.09 MB 2025-02-15 10:11:44,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29093.08 MB 2025-02-15 10:11:44,373 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:11:44,373 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:11:44,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:11:44,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:11:44,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:11:44,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:11:44,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.69 MB 2025-02-15 10:11:44,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29887.72 MB 2025-02-15 10:11:44,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:11:44,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29980.88 MB 2025-02-15 10:11:44,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38371.59 MB 2025-02-15 10:11:44,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:11:44,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29887.72 MB 2025-02-15 10:11:44,540 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:11:44,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:44,542 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:11:44,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:44,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:11:44,547 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:11:44,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:11:44,548 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:11:44,548 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:13:39,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:13:39,164 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:13:39,169 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:13:39,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:13:39,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:13:39,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:13:39,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:13:59,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:13:59,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:13:59,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.52 seconds 2025-02-15 10:13:59,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:13:59,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-15 10:13:59,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-15 10:13:59,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-15 10:13:59,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50956.60 MB 2025-02-15 10:13:59,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37998.30 MB 2025-02-15 10:13:59,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12958.30 MB 2025-02-15 10:13:59,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-15 10:13:59,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:13:59,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:13:59,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:13:59,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:13:59,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-15 10:13:59,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-15 10:13:59,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-15 10:13:59,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37998.30 MB 2025-02-15 10:13:59,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47011.86 MB 2025-02-15 10:13:59,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9013.56 MB 2025-02-15 10:13:59,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40602.49 MB 2025-02-15 10:14:01,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:14:01,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:14:01,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 10:14:01,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:01,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-15 10:14:01,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-15 10:14:01,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:14:01,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47011.86 MB 2025-02-15 10:14:01,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33254.54 MB 2025-02-15 10:14:01,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13757.32 MB 2025-02-15 10:14:01,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27253.48 MB 2025-02-15 10:14:01,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:14:01,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:14:01,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:14:01,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:01,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-15 10:14:01,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25164.47 MB 2025-02-15 10:14:01,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:14:01,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33254.54 MB 2025-02-15 10:14:01,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33254.54 MB 2025-02-15 10:14:01,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:14:01,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26581.90 MB 2025-02-15 10:14:01,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:14:01,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:14:01,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:14:01,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:01,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25164.47 MB 2025-02-15 10:14:01,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-15 10:14:01,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:14:01,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33254.54 MB 2025-02-15 10:14:01,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 10:14:01,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:14:01,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-15 10:14:01,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:14:01,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:14:01,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:14:01,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:01,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-15 10:14:01,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-15 10:14:01,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:14:01,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33254.54 MB 2025-02-15 10:14:01,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 10:14:01,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:14:01,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-15 10:14:02,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:14:02,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:14:02,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:14:02,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:02,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.87 MB 2025-02-15 10:14:02,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.87 MB 2025-02-15 10:14:02,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:14:02,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 10:14:02,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 10:14:02,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:14:02,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.66 MB 2025-02-15 10:14:02,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:14:02,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:14:02,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:14:02,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:02,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.76 MB 2025-02-15 10:14:02,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30346.53 MB 2025-02-15 10:14:02,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.77 MB 2025-02-15 10:14:02,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 10:14:02,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 10:14:02,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:14:02,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30550.65 MB 2025-02-15 10:14:02,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:14:02,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:14:02,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.96 seconds 2025-02-15 10:14:02,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:02,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-15 10:14:02,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30546.52 MB 2025-02-15 10:14:02,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12909.15 MB 2025-02-15 10:14:02,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50956.60 MB 2025-02-15 10:14:02,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 10:14:02,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15397.29 MB 2025-02-15 10:14:02,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30550.65 MB 2025-02-15 10:14:02,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:14:02,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:14:02,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:14:02,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:02,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30546.52 MB 2025-02-15 10:14:02,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22625.86 MB 2025-02-15 10:14:02,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7920.66 MB 2025-02-15 10:14:02,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 10:14:02,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-15 10:14:02,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:14:02,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33044.67 MB 2025-02-15 10:14:02,429 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-15 10:14:02,430 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:14:02,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:14:02,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:14:02,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:14:02,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:14:02,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.86 MB 2025-02-15 10:14:02,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31019.14 MB 2025-02-15 10:14:02,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-15 10:14:02,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-15 10:14:02,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39732.64 MB 2025-02-15 10:14:02,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 10:14:02,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31019.14 MB 2025-02-15 10:14:02,594 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-15 10:14:02,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:14:02,596 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:14:02,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:14:02,597 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:14:02,601 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:14:02,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:14:02,602 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:14:02,602 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:15:06,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:06,768 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:15:06,773 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:15:06,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:06,777 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2569, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:15:06,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:06,778 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2569, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:15:46,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:15:46,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:15:46,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.81 seconds 2025-02-15 10:15:46,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:46,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30871.61 MB 2025-02-15 10:15:46,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39963.16 MB 2025-02-15 10:15:46,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9091.55 MB 2025-02-15 10:15:46,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65984.79 MB 2025-02-15 10:15:46,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43476.06 MB 2025-02-15 10:15:46,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22508.73 MB 2025-02-15 10:15:46,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49054.71 MB 2025-02-15 10:15:46,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:15:46,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:15:46,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 10:15:46,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:46,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39963.16 MB 2025-02-15 10:15:46,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29135.18 MB 2025-02-15 10:15:46,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10827.98 MB 2025-02-15 10:15:46,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-15 10:15:46,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62761.47 MB 2025-02-15 10:15:46,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19285.41 MB 2025-02-15 10:15:46,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65770.09 MB 2025-02-15 10:15:48,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:15:48,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:15:48,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 10:15:48,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:48,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29135.18 MB 2025-02-15 10:15:48,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29666.02 MB 2025-02-15 10:15:48,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:15:48,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62761.47 MB 2025-02-15 10:15:48,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31960.60 MB 2025-02-15 10:15:48,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30800.87 MB 2025-02-15 10:15:48,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.57 MB 2025-02-15 10:15:48,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:15:48,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:15:48,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:15:48,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:48,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29666.02 MB 2025-02-15 10:15:48,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31555.56 MB 2025-02-15 10:15:48,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:15:48,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31960.60 MB 2025-02-15 10:15:48,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34791.75 MB 2025-02-15 10:15:48,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:15:48,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32972.99 MB 2025-02-15 10:15:49,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:15:49,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:15:49,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:15:49,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31555.56 MB 2025-02-15 10:15:49,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33797.41 MB 2025-02-15 10:15:49,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:15:49,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34791.75 MB 2025-02-15 10:15:49,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 10:15:49,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:15:49,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39341.69 MB 2025-02-15 10:15:49,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:15:49,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:15:49,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:15:49,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29666.02 MB 2025-02-15 10:15:49,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33797.41 MB 2025-02-15 10:15:49,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:15:49,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31960.60 MB 2025-02-15 10:15:49,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 10:15:49,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:15:49,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39341.69 MB 2025-02-15 10:15:49,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:15:49,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:15:49,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:15:49,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35330.95 MB 2025-02-15 10:15:49,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36097.96 MB 2025-02-15 10:15:49,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:15:49,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-15 10:15:49,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41343.25 MB 2025-02-15 10:15:49,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:15:49,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36805.75 MB 2025-02-15 10:15:49,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:15:49,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:15:49,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:15:49,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36510.85 MB 2025-02-15 10:15:49,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36739.07 MB 2025-02-15 10:15:49,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 10:15:49,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41343.25 MB 2025-02-15 10:15:49,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41343.25 MB 2025-02-15 10:15:49,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:15:49,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36955.38 MB 2025-02-15 10:15:49,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:15:49,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:15:49,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.51 seconds 2025-02-15 10:15:49,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21920.16 MB 2025-02-15 10:15:49,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36939.21 MB 2025-02-15 10:15:49,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15019.05 MB 2025-02-15 10:15:49,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57032.05 MB 2025-02-15 10:15:49,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41343.25 MB 2025-02-15 10:15:49,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15688.79 MB 2025-02-15 10:15:49,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36955.38 MB 2025-02-15 10:15:49,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:15:49,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:15:49,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:15:49,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36939.21 MB 2025-02-15 10:15:49,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26910.79 MB 2025-02-15 10:15:49,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10028.42 MB 2025-02-15 10:15:49,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41343.25 MB 2025-02-15 10:15:49,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41343.25 MB 2025-02-15 10:15:49,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:15:49,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39439.20 MB 2025-02-15 10:15:49,581 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 10:15:49,581 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:15:49,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:15:49,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:15:49,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:15:49,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:15:49,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26910.79 MB 2025-02-15 10:15:49,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35310.69 MB 2025-02-15 10:15:49,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-15 10:15:49,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41343.25 MB 2025-02-15 10:15:49,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45518.68 MB 2025-02-15 10:15:49,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 10:15:49,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35310.69 MB 2025-02-15 10:15:49,752 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 10:15:49,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:49,754 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:15:49,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:49,755 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:15:49,760 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:15:49,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:49,761 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:15:49,761 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:15:58,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:58,491 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:15:58,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:15:58,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:58,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1482, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:15:58,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:15:58,500 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1482, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:16:21,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:16:21,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:16:21,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.16 seconds 2025-02-15 10:16:21,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:21,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23295.52 MB 2025-02-15 10:16:21,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28540.50 MB 2025-02-15 10:16:21,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5244.98 MB 2025-02-15 10:16:21,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53869.54 MB 2025-02-15 10:16:21,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38654.71 MB 2025-02-15 10:16:21,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15214.84 MB 2025-02-15 10:16:21,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37523.23 MB 2025-02-15 10:16:21,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:16:21,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:16:21,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:16:21,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:21,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28540.50 MB 2025-02-15 10:16:21,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23482.31 MB 2025-02-15 10:16:21,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5058.19 MB 2025-02-15 10:16:21,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38654.71 MB 2025-02-15 10:16:21,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48481.96 MB 2025-02-15 10:16:21,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9827.25 MB 2025-02-15 10:16:21,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43317.57 MB 2025-02-15 10:16:23,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:16:23,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:16:23,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:16:23,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:23,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23482.31 MB 2025-02-15 10:16:23,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24013.15 MB 2025-02-15 10:16:23,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:16:23,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48481.96 MB 2025-02-15 10:16:23,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33409.73 MB 2025-02-15 10:16:23,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15072.23 MB 2025-02-15 10:16:23,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27991.70 MB 2025-02-15 10:16:23,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:16:23,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:16:23,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:16:23,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:23,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.15 MB 2025-02-15 10:16:23,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25902.68 MB 2025-02-15 10:16:23,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:16:23,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33409.73 MB 2025-02-15 10:16:23,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33409.73 MB 2025-02-15 10:16:23,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:23,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27320.11 MB 2025-02-15 10:16:23,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:16:23,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:16:23,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:16:23,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:23,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25902.68 MB 2025-02-15 10:16:23,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28144.54 MB 2025-02-15 10:16:23,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:16:23,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33409.73 MB 2025-02-15 10:16:23,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35769.02 MB 2025-02-15 10:16:23,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:16:23,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33688.82 MB 2025-02-15 10:16:23,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:16:23,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:16:23,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:16:23,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:23,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.15 MB 2025-02-15 10:16:23,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28144.54 MB 2025-02-15 10:16:23,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:16:23,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33409.73 MB 2025-02-15 10:16:23,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35769.02 MB 2025-02-15 10:16:23,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:16:23,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33688.82 MB 2025-02-15 10:16:24,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:16:24,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:16:24,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:16:24,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:24,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29678.08 MB 2025-02-15 10:16:24,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30445.08 MB 2025-02-15 10:16:24,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:16:24,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35769.02 MB 2025-02-15 10:16:24,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 10:16:24,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:16:24,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31152.87 MB 2025-02-15 10:16:24,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:16:24,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:16:24,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:16:24,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:24,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30857.97 MB 2025-02-15 10:16:24,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31086.58 MB 2025-02-15 10:16:24,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.60 MB 2025-02-15 10:16:24,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36186.36 MB 2025-02-15 10:16:24,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 10:16:24,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:24,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31284.20 MB 2025-02-15 10:16:24,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:16:24,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:16:24,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.60 seconds 2025-02-15 10:16:24,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:24,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18132.11 MB 2025-02-15 10:16:24,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31287.43 MB 2025-02-15 10:16:24,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13155.31 MB 2025-02-15 10:16:24,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53869.54 MB 2025-02-15 10:16:24,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 10:16:24,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17683.19 MB 2025-02-15 10:16:24,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31287.43 MB 2025-02-15 10:16:24,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:16:24,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:16:24,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:16:24,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:24,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31287.43 MB 2025-02-15 10:16:24,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23125.59 MB 2025-02-15 10:16:24,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8161.84 MB 2025-02-15 10:16:24,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36186.36 MB 2025-02-15 10:16:24,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36186.36 MB 2025-02-15 10:16:24,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:24,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33789.88 MB 2025-02-15 10:16:24,390 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 10:16:24,390 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:16:24,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:16:24,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:16:24,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:16:24,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:24,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23125.59 MB 2025-02-15 10:16:24,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31534.89 MB 2025-02-15 10:16:24,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 10:16:24,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36186.36 MB 2025-02-15 10:16:24,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44545.61 MB 2025-02-15 10:16:24,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 10:16:24,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31534.89 MB 2025-02-15 10:16:24,554 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 10:16:24,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:24,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:16:24,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:24,556 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:16:24,561 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:16:24,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:24,562 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:16:24,562 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:16:33,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:33,821 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:16:33,828 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:16:33,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:33,834 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:16:33,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:33,835 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:16:36,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:16:36,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:16:36,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.93 seconds 2025-02-15 10:16:36,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:36,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-15 10:16:36,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-15 10:16:36,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-15 10:16:36,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52904.85 MB 2025-02-15 10:16:36,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18836.62 MB 2025-02-15 10:16:36,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34068.23 MB 2025-02-15 10:16:36,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.28 MB 2025-02-15 10:16:36,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:16:36,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:16:36,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:16:36,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:36,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-15 10:16:36,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15158.53 MB 2025-02-15 10:16:36,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.53 MB 2025-02-15 10:16:36,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18836.62 MB 2025-02-15 10:16:36,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19463.67 MB 2025-02-15 10:16:36,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-15 10:16:36,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17367.80 MB 2025-02-15 10:16:37,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:16:37,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:16:37,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.88 seconds 2025-02-15 10:16:37,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15158.53 MB 2025-02-15 10:16:37,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15393.43 MB 2025-02-15 10:16:37,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 10:16:37,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19463.67 MB 2025-02-15 10:16:37,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18828.23 MB 2025-02-15 10:16:37,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -635.44 MB 2025-02-15 10:16:37,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19329.22 MB 2025-02-15 10:16:37,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:16:37,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:16:37,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:16:37,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.43 MB 2025-02-15 10:16:37,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16229.35 MB 2025-02-15 10:16:37,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 10:16:37,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18828.23 MB 2025-02-15 10:16:37,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18828.23 MB 2025-02-15 10:16:37,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:37,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16856.56 MB 2025-02-15 10:16:37,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:16:37,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:16:37,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 10:16:37,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16229.35 MB 2025-02-15 10:16:37,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17221.40 MB 2025-02-15 10:16:37,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 10:16:37,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18828.23 MB 2025-02-15 10:16:37,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21135.10 MB 2025-02-15 10:16:37,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2306.87 MB 2025-02-15 10:16:37,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19675.63 MB 2025-02-15 10:16:37,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:16:37,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:16:37,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:16:37,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.43 MB 2025-02-15 10:16:37,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17221.40 MB 2025-02-15 10:16:37,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 10:16:37,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18828.23 MB 2025-02-15 10:16:37,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21135.10 MB 2025-02-15 10:16:37,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2306.87 MB 2025-02-15 10:16:37,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19675.63 MB 2025-02-15 10:16:37,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:16:37,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:16:37,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:16:37,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17900.93 MB 2025-02-15 10:16:37,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18241.25 MB 2025-02-15 10:16:37,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 10:16:37,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21135.10 MB 2025-02-15 10:16:37,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21317.55 MB 2025-02-15 10:16:37,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 10:16:37,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18558.94 MB 2025-02-15 10:16:37,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:16:37,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:16:37,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:16:37,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18423.96 MB 2025-02-15 10:16:37,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18653.89 MB 2025-02-15 10:16:37,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.93 MB 2025-02-15 10:16:37,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21317.55 MB 2025-02-15 10:16:37,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21317.55 MB 2025-02-15 10:16:37,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:37,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18694.47 MB 2025-02-15 10:16:37,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:16:37,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:16:37,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.16 seconds 2025-02-15 10:16:37,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:37,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-15 10:16:37,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18854.96 MB 2025-02-15 10:16:37,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5252.15 MB 2025-02-15 10:16:37,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52904.85 MB 2025-02-15 10:16:37,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21317.55 MB 2025-02-15 10:16:37,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31587.30 MB 2025-02-15 10:16:37,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18854.96 MB 2025-02-15 10:16:38,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:16:38,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:16:38,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:16:38,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:38,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18854.96 MB 2025-02-15 10:16:38,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17556.65 MB 2025-02-15 10:16:38,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1298.31 MB 2025-02-15 10:16:38,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21317.55 MB 2025-02-15 10:16:38,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21317.55 MB 2025-02-15 10:16:38,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:16:38,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19090.04 MB 2025-02-15 10:16:38,309 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:16:38,309 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 10:16:38,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:16:38,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:16:38,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:16:38,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:16:38,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17556.65 MB 2025-02-15 10:16:38,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25995.67 MB 2025-02-15 10:16:38,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:16:38,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21317.55 MB 2025-02-15 10:16:38,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31807.50 MB 2025-02-15 10:16:38,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:16:38,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25995.67 MB 2025-02-15 10:16:38,568 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:16:38,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:38,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:16:38,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:38,573 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:16:38,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:16:38,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:16:38,591 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:16:38,591 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 10:17:01,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:01,541 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:17:01,546 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:17:01,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:01,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:17:01,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:01,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:17:04,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:17:04,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:17:04,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-15 10:17:04,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:04,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-15 10:17:04,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-15 10:17:04,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-15 10:17:04,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44392.51 MB 2025-02-15 10:17:04,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18675.14 MB 2025-02-15 10:17:04,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25717.37 MB 2025-02-15 10:17:04,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-15 10:17:04,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:17:04,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:17:04,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:17:04,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:04,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-15 10:17:04,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.51 MB 2025-02-15 10:17:04,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.02 MB 2025-02-15 10:17:04,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18675.14 MB 2025-02-15 10:17:04,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18675.14 MB 2025-02-15 10:17:04,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:04,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17216.25 MB 2025-02-15 10:17:05,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:17:05,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:17:05,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 10:17:05,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.51 MB 2025-02-15 10:17:05,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15304.14 MB 2025-02-15 10:17:05,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-15 10:17:05,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18675.14 MB 2025-02-15 10:17:05,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17144.22 MB 2025-02-15 10:17:05,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1530.92 MB 2025-02-15 10:17:05,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19253.20 MB 2025-02-15 10:17:05,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:17:05,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:17:05,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:17:05,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.07 MB 2025-02-15 10:17:05,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16093.03 MB 2025-02-15 10:17:05,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.95 MB 2025-02-15 10:17:05,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17144.22 MB 2025-02-15 10:17:05,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17932.75 MB 2025-02-15 10:17:05,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 788.53 MB 2025-02-15 10:17:05,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16685.07 MB 2025-02-15 10:17:05,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:17:05,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:17:05,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:17:05,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16093.03 MB 2025-02-15 10:17:05,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17029.96 MB 2025-02-15 10:17:05,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.93 MB 2025-02-15 10:17:05,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17932.75 MB 2025-02-15 10:17:05,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20298.33 MB 2025-02-15 10:17:05,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2365.59 MB 2025-02-15 10:17:05,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19345.97 MB 2025-02-15 10:17:05,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:17:05,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:17:05,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:17:05,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.07 MB 2025-02-15 10:17:05,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17029.96 MB 2025-02-15 10:17:05,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1725.88 MB 2025-02-15 10:17:05,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17144.22 MB 2025-02-15 10:17:05,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20298.33 MB 2025-02-15 10:17:05,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3154.12 MB 2025-02-15 10:17:05,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19345.97 MB 2025-02-15 10:17:05,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:17:05,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:17:05,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:17:05,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17670.21 MB 2025-02-15 10:17:05,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17990.69 MB 2025-02-15 10:17:05,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.49 MB 2025-02-15 10:17:05,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20298.33 MB 2025-02-15 10:17:05,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20468.20 MB 2025-02-15 10:17:05,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 10:17:05,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18294.35 MB 2025-02-15 10:17:05,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:17:05,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:17:05,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:17:05,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18163.08 MB 2025-02-15 10:17:05,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18388.35 MB 2025-02-15 10:17:05,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.26 MB 2025-02-15 10:17:05,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20468.20 MB 2025-02-15 10:17:05,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20468.20 MB 2025-02-15 10:17:05,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:05,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18407.64 MB 2025-02-15 10:17:05,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:17:05,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:17:05,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.87 seconds 2025-02-15 10:17:05,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-15 10:17:05,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18589.35 MB 2025-02-15 10:17:05,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4990.02 MB 2025-02-15 10:17:05,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44392.51 MB 2025-02-15 10:17:05,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20468.20 MB 2025-02-15 10:17:05,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23924.31 MB 2025-02-15 10:17:05,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18589.35 MB 2025-02-15 10:17:05,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:17:05,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:17:05,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:17:05,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18589.35 MB 2025-02-15 10:17:05,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17502.85 MB 2025-02-15 10:17:05,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1086.49 MB 2025-02-15 10:17:05,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20468.20 MB 2025-02-15 10:17:05,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20468.20 MB 2025-02-15 10:17:05,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:05,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19191.93 MB 2025-02-15 10:17:05,711 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 10:17:05,712 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:17:05,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:17:05,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:17:05,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:17:05,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:05,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17502.85 MB 2025-02-15 10:17:05,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25938.45 MB 2025-02-15 10:17:05,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 10:17:05,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20468.20 MB 2025-02-15 10:17:05,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30953.96 MB 2025-02-15 10:17:05,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 10:17:05,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25938.45 MB 2025-02-15 10:17:05,878 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 10:17:05,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:05,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:17:05,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:05,881 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:17:05,885 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:17:05,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:05,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:17:05,887 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:17:30,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:30,759 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:17:30,764 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:17:30,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:30,769 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 657, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:17:30,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:30,771 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 657, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:17:40,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:17:40,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:17:40,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.20 seconds 2025-02-15 10:17:40,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:40,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.79 MB 2025-02-15 10:17:40,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19872.53 MB 2025-02-15 10:17:40,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2325.74 MB 2025-02-15 10:17:40,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39342.57 MB 2025-02-15 10:17:40,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25094.52 MB 2025-02-15 10:17:40,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14248.05 MB 2025-02-15 10:17:40,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28830.10 MB 2025-02-15 10:17:41,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:17:41,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:17:41,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:17:41,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:41,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19872.53 MB 2025-02-15 10:17:41,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19194.44 MB 2025-02-15 10:17:41,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -678.09 MB 2025-02-15 10:17:41,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25094.52 MB 2025-02-15 10:17:41,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30037.51 MB 2025-02-15 10:17:41,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4942.99 MB 2025-02-15 10:17:41,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27863.75 MB 2025-02-15 10:17:42,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:17:42,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:17:42,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:17:42,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:42,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19194.44 MB 2025-02-15 10:17:42,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19725.28 MB 2025-02-15 10:17:42,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:17:42,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30037.51 MB 2025-02-15 10:17:42,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24893.19 MB 2025-02-15 10:17:42,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5144.31 MB 2025-02-15 10:17:42,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23703.83 MB 2025-02-15 10:17:42,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:17:42,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:17:42,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:17:42,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:42,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19725.28 MB 2025-02-15 10:17:42,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21614.81 MB 2025-02-15 10:17:42,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:17:42,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 10:17:42,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24893.19 MB 2025-02-15 10:17:42,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:42,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23032.24 MB 2025-02-15 10:17:43,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:17:43,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:17:43,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:17:43,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21614.81 MB 2025-02-15 10:17:43,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23857.72 MB 2025-02-15 10:17:43,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 10:17:43,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 10:17:43,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31501.32 MB 2025-02-15 10:17:43,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 10:17:43,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29402.00 MB 2025-02-15 10:17:43,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:17:43,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:17:43,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:17:43,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19725.28 MB 2025-02-15 10:17:43,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23857.72 MB 2025-02-15 10:17:43,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 10:17:43,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24893.19 MB 2025-02-15 10:17:43,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31501.32 MB 2025-02-15 10:17:43,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 10:17:43,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29402.00 MB 2025-02-15 10:17:43,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:17:43,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:17:43,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:17:43,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25391.26 MB 2025-02-15 10:17:43,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26158.26 MB 2025-02-15 10:17:43,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:17:43,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31501.32 MB 2025-02-15 10:17:43,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 10:17:43,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:17:43,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26866.05 MB 2025-02-15 10:17:43,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:17:43,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:17:43,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:17:43,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26571.15 MB 2025-02-15 10:17:43,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26800.47 MB 2025-02-15 10:17:43,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.32 MB 2025-02-15 10:17:43,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 10:17:43,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 10:17:43,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:43,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.01 MB 2025-02-15 10:17:43,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:17:43,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:17:43,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.61 seconds 2025-02-15 10:17:43,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15257.75 MB 2025-02-15 10:17:43,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27001.54 MB 2025-02-15 10:17:43,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11743.79 MB 2025-02-15 10:17:43,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39342.57 MB 2025-02-15 10:17:43,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 10:17:43,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7423.92 MB 2025-02-15 10:17:43,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27001.54 MB 2025-02-15 10:17:43,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:17:43,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:17:43,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:17:43,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27001.54 MB 2025-02-15 10:17:43,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20262.14 MB 2025-02-15 10:17:43,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6739.40 MB 2025-02-15 10:17:43,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 10:17:43,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31918.65 MB 2025-02-15 10:17:43,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:17:43,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29513.21 MB 2025-02-15 10:17:43,676 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:17:43,676 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:17:43,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:17:43,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:17:43,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:17:43,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:17:43,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20262.14 MB 2025-02-15 10:17:43,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28701.16 MB 2025-02-15 10:17:43,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:17:43,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31918.65 MB 2025-02-15 10:17:43,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40309.36 MB 2025-02-15 10:17:43,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:17:43,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28701.16 MB 2025-02-15 10:17:43,848 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:17:43,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:43,850 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:17:43,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:43,851 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:17:43,856 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:17:43,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:17:43,857 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:17:43,857 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:18:42,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:42,315 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:18:42,320 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:18:42,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:42,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 535, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:18:42,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:42,325 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 535, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:18:50,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:18:50,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:18:50,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.28 seconds 2025-02-15 10:18:50,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:50,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16696.67 MB 2025-02-15 10:18:50,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18590.01 MB 2025-02-15 10:18:50,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1893.34 MB 2025-02-15 10:18:50,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52894.37 MB 2025-02-15 10:18:50,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20461.91 MB 2025-02-15 10:18:50,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32432.46 MB 2025-02-15 10:18:50,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27527.00 MB 2025-02-15 10:18:50,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:18:50,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:18:50,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 10:18:50,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:50,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18590.01 MB 2025-02-15 10:18:50,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18560.20 MB 2025-02-15 10:18:50,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -29.81 MB 2025-02-15 10:18:50,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20461.91 MB 2025-02-15 10:18:50,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25933.38 MB 2025-02-15 10:18:50,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5471.47 MB 2025-02-15 10:18:50,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26159.59 MB 2025-02-15 10:18:52,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:18:52,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:18:52,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:18:52,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:52,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18560.20 MB 2025-02-15 10:18:52,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19091.04 MB 2025-02-15 10:18:52,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:18:52,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25933.38 MB 2025-02-15 10:18:52,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22586.33 MB 2025-02-15 10:18:52,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3347.05 MB 2025-02-15 10:18:52,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23070.62 MB 2025-02-15 10:18:52,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:18:52,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:18:52,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:18:52,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:52,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19091.04 MB 2025-02-15 10:18:52,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20980.57 MB 2025-02-15 10:18:52,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:18:52,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22586.33 MB 2025-02-15 10:18:52,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24473.76 MB 2025-02-15 10:18:52,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:18:52,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22398.00 MB 2025-02-15 10:18:52,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:18:52,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:18:52,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:18:52,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:52,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20980.57 MB 2025-02-15 10:18:52,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23223.48 MB 2025-02-15 10:18:52,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 10:18:52,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24473.76 MB 2025-02-15 10:18:52,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30844.91 MB 2025-02-15 10:18:52,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 10:18:52,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28767.76 MB 2025-02-15 10:18:52,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:18:52,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:18:52,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:18:52,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:52,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19091.04 MB 2025-02-15 10:18:52,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23223.48 MB 2025-02-15 10:18:52,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 10:18:52,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22586.33 MB 2025-02-15 10:18:52,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30844.91 MB 2025-02-15 10:18:52,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8258.58 MB 2025-02-15 10:18:52,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28767.76 MB 2025-02-15 10:18:52,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:18:52,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:18:52,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:18:52,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:52,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24757.02 MB 2025-02-15 10:18:52,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25524.02 MB 2025-02-15 10:18:52,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:18:52,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30844.91 MB 2025-02-15 10:18:52,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31262.24 MB 2025-02-15 10:18:52,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:18:52,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26231.81 MB 2025-02-15 10:18:53,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:18:53,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:18:53,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:18:53,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:53,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25936.91 MB 2025-02-15 10:18:53,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26166.62 MB 2025-02-15 10:18:53,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.71 MB 2025-02-15 10:18:53,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31262.24 MB 2025-02-15 10:18:53,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31262.24 MB 2025-02-15 10:18:53,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:18:53,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26329.88 MB 2025-02-15 10:18:53,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:18:53,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:18:53,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.68 seconds 2025-02-15 10:18:53,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:53,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14832.69 MB 2025-02-15 10:18:53,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26367.69 MB 2025-02-15 10:18:53,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11535.01 MB 2025-02-15 10:18:53,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52894.37 MB 2025-02-15 10:18:53,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31262.24 MB 2025-02-15 10:18:53,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21632.12 MB 2025-02-15 10:18:53,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26367.69 MB 2025-02-15 10:18:53,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:18:53,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:18:53,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:18:53,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:53,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26367.69 MB 2025-02-15 10:18:53,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19837.08 MB 2025-02-15 10:18:53,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6530.62 MB 2025-02-15 10:18:53,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31262.24 MB 2025-02-15 10:18:53,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31262.24 MB 2025-02-15 10:18:53,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:18:53,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28879.36 MB 2025-02-15 10:18:53,294 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:18:53,294 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 10:18:53,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:18:53,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:18:53,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:18:53,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:18:53,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19837.08 MB 2025-02-15 10:18:53,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28276.10 MB 2025-02-15 10:18:53,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:18:53,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31262.24 MB 2025-02-15 10:18:53,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41752.20 MB 2025-02-15 10:18:53,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:18:53,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28276.10 MB 2025-02-15 10:18:53,461 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:18:53,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:53,463 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:18:53,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:53,464 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:18:53,468 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:18:53,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:18:53,469 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:18:53,470 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 10:19:02,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:02,713 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:19:02,721 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:19:02,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:02,727 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:19:02,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:02,729 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:19:28,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:19:28,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:19:28,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.77 seconds 2025-02-15 10:19:28,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:28,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24466.17 MB 2025-02-15 10:19:28,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30305.43 MB 2025-02-15 10:19:28,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5839.26 MB 2025-02-15 10:19:28,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54337.21 MB 2025-02-15 10:19:28,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39099.30 MB 2025-02-15 10:19:28,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15237.91 MB 2025-02-15 10:19:28,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39146.87 MB 2025-02-15 10:19:28,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:19:28,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:19:28,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 10:19:28,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:28,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30305.43 MB 2025-02-15 10:19:28,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24355.69 MB 2025-02-15 10:19:28,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5949.74 MB 2025-02-15 10:19:28,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39099.30 MB 2025-02-15 10:19:28,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51952.75 MB 2025-02-15 10:19:28,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12853.44 MB 2025-02-15 10:19:28,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47501.90 MB 2025-02-15 10:19:30,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:19:30,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:19:30,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 10:19:30,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:30,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24355.69 MB 2025-02-15 10:19:30,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24886.53 MB 2025-02-15 10:19:30,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:19:30,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51952.75 MB 2025-02-15 10:19:30,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:19:30,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17278.44 MB 2025-02-15 10:19:30,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28865.08 MB 2025-02-15 10:19:30,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:19:30,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:19:30,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:19:30,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:30,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-15 10:19:30,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26776.06 MB 2025-02-15 10:19:30,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:19:30,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:19:30,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:19:30,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:19:30,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28193.49 MB 2025-02-15 10:19:30,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:19:30,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:19:30,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:19:30,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:30,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26776.06 MB 2025-02-15 10:19:30,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-15 10:19:30,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:19:30,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:19:30,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37033.61 MB 2025-02-15 10:19:30,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:19:30,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-15 10:19:30,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:19:30,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:19:30,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:19:30,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:30,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-15 10:19:30,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-15 10:19:30,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:19:30,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:19:30,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37033.61 MB 2025-02-15 10:19:30,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:19:30,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-15 10:19:30,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:19:30,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:19:30,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:19:30,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:30,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30551.46 MB 2025-02-15 10:19:30,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31318.46 MB 2025-02-15 10:19:30,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:19:30,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37033.61 MB 2025-02-15 10:19:30,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37448.84 MB 2025-02-15 10:19:30,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:19:30,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32026.25 MB 2025-02-15 10:19:31,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:19:31,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:19:31,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:19:31,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:31,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31731.35 MB 2025-02-15 10:19:31,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31960.46 MB 2025-02-15 10:19:31,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-15 10:19:31,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37448.84 MB 2025-02-15 10:19:31,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37448.84 MB 2025-02-15 10:19:31,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:19:31,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32203.19 MB 2025-02-15 10:19:31,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:19:31,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:19:31,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.28 seconds 2025-02-15 10:19:31,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:31,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18717.44 MB 2025-02-15 10:19:31,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32161.48 MB 2025-02-15 10:19:31,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13444.04 MB 2025-02-15 10:19:31,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54337.21 MB 2025-02-15 10:19:31,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37448.84 MB 2025-02-15 10:19:31,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16888.37 MB 2025-02-15 10:19:31,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32203.19 MB 2025-02-15 10:19:31,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:19:31,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:19:31,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:19:31,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:31,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32161.48 MB 2025-02-15 10:19:31,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23721.07 MB 2025-02-15 10:19:31,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8440.42 MB 2025-02-15 10:19:31,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37448.84 MB 2025-02-15 10:19:31,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37448.84 MB 2025-02-15 10:19:31,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:19:31,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34672.54 MB 2025-02-15 10:19:31,308 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 10:19:31,308 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:19:31,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:19:31,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:19:31,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:19:31,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:19:31,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23721.07 MB 2025-02-15 10:19:31,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32158.54 MB 2025-02-15 10:19:31,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 10:19:31,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37448.84 MB 2025-02-15 10:19:31,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45837.45 MB 2025-02-15 10:19:31,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 10:19:31,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32158.54 MB 2025-02-15 10:19:31,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 10:19:31,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:31,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:19:31,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:31,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:19:31,482 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:19:31,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:19:31,483 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:19:31,483 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:20:21,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:21,673 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:20:21,678 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:20:21,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:21,682 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:20:21,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:21,683 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:20:23,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:20:23,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:20:23,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.06 seconds 2025-02-15 10:20:23,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:23,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13875.05 MB 2025-02-15 10:20:23,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14335.11 MB 2025-02-15 10:20:23,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 460.06 MB 2025-02-15 10:20:23,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54226.06 MB 2025-02-15 10:20:23,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:23,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32134.66 MB 2025-02-15 10:20:23,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23346.42 MB 2025-02-15 10:20:23,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:20:23,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:20:23,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:20:23,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:23,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14335.11 MB 2025-02-15 10:20:23,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14558.07 MB 2025-02-15 10:20:23,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.95 MB 2025-02-15 10:20:23,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:23,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:23,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:23,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16161.22 MB 2025-02-15 10:20:24,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:20:24,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:20:24,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-15 10:20:24,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14558.07 MB 2025-02-15 10:20:24,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14730.59 MB 2025-02-15 10:20:24,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-15 10:20:24,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:24,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:24,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:24,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18727.72 MB 2025-02-15 10:20:24,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:20:24,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:20:24,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:20:24,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14730.52 MB 2025-02-15 10:20:24,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15344.47 MB 2025-02-15 10:20:24,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-15 10:20:24,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:24,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:24,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:24,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15805.14 MB 2025-02-15 10:20:24,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:20:24,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:20:24,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:20:24,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15344.47 MB 2025-02-15 10:20:24,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16073.12 MB 2025-02-15 10:20:24,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-15 10:20:24,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:24,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:24,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:24,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.97 MB 2025-02-15 10:20:24,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:20:24,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:20:24,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:20:24,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14730.52 MB 2025-02-15 10:20:24,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16073.12 MB 2025-02-15 10:20:24,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-15 10:20:24,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:24,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22091.40 MB 2025-02-15 10:20:24,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:24,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.97 MB 2025-02-15 10:20:24,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:20:24,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:20:24,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 10:20:24,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16571.52 MB 2025-02-15 10:20:24,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16820.80 MB 2025-02-15 10:20:24,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-15 10:20:24,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22091.40 MB 2025-02-15 10:20:24,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22221.42 MB 2025-02-15 10:20:24,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-15 10:20:24,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17062.12 MB 2025-02-15 10:20:24,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:20:24,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:20:24,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:20:24,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16955.00 MB 2025-02-15 10:20:24,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17151.50 MB 2025-02-15 10:20:24,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.50 MB 2025-02-15 10:20:24,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22221.42 MB 2025-02-15 10:20:24,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22225.62 MB 2025-02-15 10:20:24,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 10:20:24,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17151.50 MB 2025-02-15 10:20:24,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:20:24,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:20:24,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.86 seconds 2025-02-15 10:20:24,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13422.12 MB 2025-02-15 10:20:24,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17343.67 MB 2025-02-15 10:20:24,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3921.55 MB 2025-02-15 10:20:24,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54226.06 MB 2025-02-15 10:20:24,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22225.62 MB 2025-02-15 10:20:24,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32000.44 MB 2025-02-15 10:20:24,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17343.67 MB 2025-02-15 10:20:24,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:20:24,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:20:24,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 10:20:24,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17343.67 MB 2025-02-15 10:20:24,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17014.40 MB 2025-02-15 10:20:24,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -329.27 MB 2025-02-15 10:20:24,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22225.62 MB 2025-02-15 10:20:24,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22225.62 MB 2025-02-15 10:20:24,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:24,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18687.93 MB 2025-02-15 10:20:24,822 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7800, cut from 7802 2025-02-15 10:20:24,822 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:20:24,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:20:24,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:20:24,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:20:24,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:24,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17014.40 MB 2025-02-15 10:20:24,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25080.42 MB 2025-02-15 10:20:24,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8066.02 MB 2025-02-15 10:20:24,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22225.62 MB 2025-02-15 10:20:24,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30245.13 MB 2025-02-15 10:20:24,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8019.51 MB 2025-02-15 10:20:24,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25080.42 MB 2025-02-15 10:20:24,986 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7592] 2025-02-15 10:20:24,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:24,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:20:24,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:24,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:20:24,993 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:20:24,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:24,994 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:20:24,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:20:34,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:34,142 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:20:34,150 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:20:34,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:34,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:20:34,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:34,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:20:54,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:20:54,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:20:54,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.38 seconds 2025-02-15 10:20:54,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:54,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22076.09 MB 2025-02-15 10:20:54,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26702.41 MB 2025-02-15 10:20:54,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4626.32 MB 2025-02-15 10:20:54,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38264.64 MB 2025-02-15 10:20:54,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37327.21 MB 2025-02-15 10:20:54,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -937.43 MB 2025-02-15 10:20:54,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35624.33 MB 2025-02-15 10:20:54,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:20:54,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:20:54,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:20:54,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:54,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26702.41 MB 2025-02-15 10:20:54,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22572.54 MB 2025-02-15 10:20:54,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4129.87 MB 2025-02-15 10:20:54,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37327.21 MB 2025-02-15 10:20:54,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46508.54 MB 2025-02-15 10:20:54,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9181.33 MB 2025-02-15 10:20:54,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40509.18 MB 2025-02-15 10:20:56,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:20:56,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:20:56,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:20:56,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22572.54 MB 2025-02-15 10:20:56,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23103.38 MB 2025-02-15 10:20:56,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:20:56,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46508.54 MB 2025-02-15 10:20:56,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32700.89 MB 2025-02-15 10:20:56,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13807.65 MB 2025-02-15 10:20:56,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27081.92 MB 2025-02-15 10:20:56,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:20:56,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:20:56,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:20:56,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-15 10:20:56,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24992.91 MB 2025-02-15 10:20:56,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:20:56,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32700.89 MB 2025-02-15 10:20:56,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32700.89 MB 2025-02-15 10:20:56,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:56,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26410.34 MB 2025-02-15 10:20:56,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:20:56,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:20:56,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:20:56,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24992.91 MB 2025-02-15 10:20:56,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-15 10:20:56,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:20:56,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32700.89 MB 2025-02-15 10:20:56,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-15 10:20:56,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:20:56,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-15 10:20:56,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:20:56,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:20:56,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:20:56,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-15 10:20:56,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-15 10:20:56,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:20:56,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32700.89 MB 2025-02-15 10:20:56,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-15 10:20:56,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:20:56,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-15 10:20:56,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:20:56,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:20:56,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:20:56,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28768.31 MB 2025-02-15 10:20:56,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29535.31 MB 2025-02-15 10:20:56,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:20:56,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35532.05 MB 2025-02-15 10:20:56,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35949.38 MB 2025-02-15 10:20:56,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:20:56,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30243.10 MB 2025-02-15 10:20:56,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:20:56,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:20:56,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:20:56,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29948.20 MB 2025-02-15 10:20:56,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30175.22 MB 2025-02-15 10:20:56,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.02 MB 2025-02-15 10:20:56,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35949.38 MB 2025-02-15 10:20:56,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35949.38 MB 2025-02-15 10:20:56,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:56,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.01 MB 2025-02-15 10:20:56,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:20:56,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:20:56,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.81 seconds 2025-02-15 10:20:56,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:56,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17522.40 MB 2025-02-15 10:20:56,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30375.46 MB 2025-02-15 10:20:56,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12853.06 MB 2025-02-15 10:20:56,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38264.64 MB 2025-02-15 10:20:56,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35949.38 MB 2025-02-15 10:20:56,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2315.26 MB 2025-02-15 10:20:56,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.01 MB 2025-02-15 10:20:57,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:20:57,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:20:57,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:20:57,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:57,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30375.46 MB 2025-02-15 10:20:57,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22514.45 MB 2025-02-15 10:20:57,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7861.01 MB 2025-02-15 10:20:57,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35949.38 MB 2025-02-15 10:20:57,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35949.38 MB 2025-02-15 10:20:57,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:20:57,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32876.68 MB 2025-02-15 10:20:57,259 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 10:20:57,259 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:20:57,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:20:57,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:20:57,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:20:57,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:20:57,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22514.45 MB 2025-02-15 10:20:57,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30919.53 MB 2025-02-15 10:20:57,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-15 10:20:57,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35949.38 MB 2025-02-15 10:20:57,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44304.43 MB 2025-02-15 10:20:57,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 10:20:57,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30919.53 MB 2025-02-15 10:20:57,424 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 10:20:57,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:57,425 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:20:57,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:57,426 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:20:57,431 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:20:57,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:20:57,432 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:20:57,432 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:22:12,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:12,418 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:22:12,423 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:22:12,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:12,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 146, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:22:12,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:12,428 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 146, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:22:14,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:22:14,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:22:14,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.28 seconds 2025-02-15 10:22:14,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13986.06 MB 2025-02-15 10:22:14,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14502.74 MB 2025-02-15 10:22:14,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.69 MB 2025-02-15 10:22:14,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52659.49 MB 2025-02-15 10:22:14,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35238.45 MB 2025-02-15 10:22:14,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23457.43 MB 2025-02-15 10:22:14,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:22:14,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:22:14,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:22:14,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14502.74 MB 2025-02-15 10:22:14,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13910.31 MB 2025-02-15 10:22:14,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -592.43 MB 2025-02-15 10:22:14,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14868.00 MB 2025-02-15 10:22:14,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:22:14,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:22:14,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:22:14,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13910.31 MB 2025-02-15 10:22:14,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13944.82 MB 2025-02-15 10:22:14,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 34.50 MB 2025-02-15 10:22:14,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15569.73 MB 2025-02-15 10:22:14,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:22:14,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:22:14,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:22:14,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.75 MB 2025-02-15 10:22:14,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14067.54 MB 2025-02-15 10:22:14,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 122.79 MB 2025-02-15 10:22:14,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14159.68 MB 2025-02-15 10:22:14,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:22:14,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:22:14,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:22:14,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14067.54 MB 2025-02-15 10:22:14,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14213.32 MB 2025-02-15 10:22:14,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 145.78 MB 2025-02-15 10:22:14,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.64 MB 2025-02-15 10:22:14,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:22:14,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:22:14,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:22:14,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.75 MB 2025-02-15 10:22:14,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14213.32 MB 2025-02-15 10:22:14,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.57 MB 2025-02-15 10:22:14,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 10:22:14,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.64 MB 2025-02-15 10:22:14,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:22:14,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:22:14,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:22:14,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.00 MB 2025-02-15 10:22:14,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14362.86 MB 2025-02-15 10:22:14,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 49.86 MB 2025-02-15 10:22:14,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 10:22:14,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17446.21 MB 2025-02-15 10:22:14,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25.17 MB 2025-02-15 10:22:14,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14428.00 MB 2025-02-15 10:22:14,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:22:14,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:22:14,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:22:14,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14389.71 MB 2025-02-15 10:22:14,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14421.50 MB 2025-02-15 10:22:14,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 31.79 MB 2025-02-15 10:22:14,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17446.21 MB 2025-02-15 10:22:14,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17446.21 MB 2025-02-15 10:22:14,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:14,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14421.50 MB 2025-02-15 10:22:14,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:22:14,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:22:14,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-15 10:22:14,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:14,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13477.38 MB 2025-02-15 10:22:14,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14481.04 MB 2025-02-15 10:22:14,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.65 MB 2025-02-15 10:22:14,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52659.49 MB 2025-02-15 10:22:14,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17446.21 MB 2025-02-15 10:22:14,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35213.28 MB 2025-02-15 10:22:14,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14481.04 MB 2025-02-15 10:22:15,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:22:15,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:22:15,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:22:15,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:15,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14481.04 MB 2025-02-15 10:22:15,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15373.52 MB 2025-02-15 10:22:15,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 892.49 MB 2025-02-15 10:22:15,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17446.21 MB 2025-02-15 10:22:15,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17448.30 MB 2025-02-15 10:22:15,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 10:22:15,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15462.76 MB 2025-02-15 10:22:15,027 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 2407, cut from 2409 2025-02-15 10:22:15,027 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-15 10:22:15,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:22:15,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:22:15,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:22:15,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:15,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14524.36 MB 2025-02-15 10:22:15,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17023.24 MB 2025-02-15 10:22:15,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2498.89 MB 2025-02-15 10:22:15,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17448.30 MB 2025-02-15 10:22:15,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18691.92 MB 2025-02-15 10:22:15,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1243.61 MB 2025-02-15 10:22:15,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17023.24 MB 2025-02-15 10:22:15,078 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 2199] 2025-02-15 10:22:15,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:15,079 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:22:15,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:15,080 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:22:15,085 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:22:15,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:15,086 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:22:15,086 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-15 10:22:25,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:25,466 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:22:25,471 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:22:25,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:25,474 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:22:25,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:25,475 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:22:52,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:22:52,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:22:52,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.13 seconds 2025-02-15 10:22:52,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:52,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25191.12 MB 2025-02-15 10:22:52,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31398.69 MB 2025-02-15 10:22:52,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-15 10:22:52,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33405.53 MB 2025-02-15 10:22:52,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34911.29 MB 2025-02-15 10:22:52,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1505.76 MB 2025-02-15 10:22:52,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40324.80 MB 2025-02-15 10:22:52,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:22:52,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:22:52,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:22:52,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:52,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31398.69 MB 2025-02-15 10:22:52,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24897.53 MB 2025-02-15 10:22:52,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6501.16 MB 2025-02-15 10:22:52,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34911.29 MB 2025-02-15 10:22:52,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49280.97 MB 2025-02-15 10:22:52,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14369.69 MB 2025-02-15 10:22:52,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49809.24 MB 2025-02-15 10:22:54,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:22:54,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:22:54,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:22:54,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:54,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24897.53 MB 2025-02-15 10:22:54,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25428.37 MB 2025-02-15 10:22:54,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:22:54,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49280.97 MB 2025-02-15 10:22:54,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27724.35 MB 2025-02-15 10:22:54,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21556.63 MB 2025-02-15 10:22:54,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29406.92 MB 2025-02-15 10:22:54,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:22:54,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:22:54,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:22:54,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:54,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25428.37 MB 2025-02-15 10:22:54,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27317.80 MB 2025-02-15 10:22:54,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.43 MB 2025-02-15 10:22:54,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27724.35 MB 2025-02-15 10:22:54,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30555.50 MB 2025-02-15 10:22:54,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:22:54,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28735.23 MB 2025-02-15 10:22:54,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:22:54,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:22:54,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 10:22:54,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:54,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27317.80 MB 2025-02-15 10:22:54,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29559.65 MB 2025-02-15 10:22:54,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:22:54,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30555.50 MB 2025-02-15 10:22:54,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-15 10:22:54,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:22:54,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35103.94 MB 2025-02-15 10:22:54,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:22:54,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:22:54,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:22:54,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:54,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25428.37 MB 2025-02-15 10:22:54,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29559.65 MB 2025-02-15 10:22:54,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.28 MB 2025-02-15 10:22:54,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27724.35 MB 2025-02-15 10:22:54,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-15 10:22:54,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:22:54,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35103.94 MB 2025-02-15 10:22:55,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:22:55,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:22:55,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:22:55,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:55,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31093.20 MB 2025-02-15 10:22:55,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31860.20 MB 2025-02-15 10:22:55,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:22:55,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36689.67 MB 2025-02-15 10:22:55,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37107.01 MB 2025-02-15 10:22:55,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:22:55,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32567.99 MB 2025-02-15 10:22:55,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:22:55,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:22:55,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:22:55,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:55,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32273.09 MB 2025-02-15 10:22:55,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32501.18 MB 2025-02-15 10:22:55,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.09 MB 2025-02-15 10:22:55,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37107.01 MB 2025-02-15 10:22:55,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37107.01 MB 2025-02-15 10:22:55,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:55,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32745.34 MB 2025-02-15 10:22:55,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:22:55,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:22:55,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.64 seconds 2025-02-15 10:22:55,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:55,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19079.91 MB 2025-02-15 10:22:55,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32701.29 MB 2025-02-15 10:22:55,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13621.38 MB 2025-02-15 10:22:55,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27292.34 MB 2025-02-15 10:22:55,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37107.01 MB 2025-02-15 10:22:55,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9814.67 MB 2025-02-15 10:22:55,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32745.34 MB 2025-02-15 10:22:55,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:22:55,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:22:55,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:22:55,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:55,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32701.29 MB 2025-02-15 10:22:55,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24070.08 MB 2025-02-15 10:22:55,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8631.22 MB 2025-02-15 10:22:55,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37107.01 MB 2025-02-15 10:22:55,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37107.01 MB 2025-02-15 10:22:55,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:22:55,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35200.98 MB 2025-02-15 10:22:55,421 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 10:22:55,422 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:22:55,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:22:55,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:22:55,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:22:55,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:22:55,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24070.08 MB 2025-02-15 10:22:55,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32469.47 MB 2025-02-15 10:22:55,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-15 10:22:55,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37107.01 MB 2025-02-15 10:22:55,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45457.87 MB 2025-02-15 10:22:55,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 10:22:55,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32469.47 MB 2025-02-15 10:22:55,592 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 10:22:55,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:55,594 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:22:55,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:55,595 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:22:55,599 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:22:55,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:22:55,601 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:22:55,601 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:23:41,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:41,103 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:23:41,108 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:23:41,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:41,112 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 220, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:23:41,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:41,113 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 220, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:23:44,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:23:44,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:23:44,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.41 seconds 2025-02-15 10:23:44,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:44,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14501.70 MB 2025-02-15 10:23:44,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15280.27 MB 2025-02-15 10:23:44,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 778.57 MB 2025-02-15 10:23:44,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53808.73 MB 2025-02-15 10:23:44,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 10:23:44,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35204.89 MB 2025-02-15 10:23:44,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24199.56 MB 2025-02-15 10:23:44,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:23:44,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:23:44,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:23:44,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:44,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15280.27 MB 2025-02-15 10:23:44,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15657.42 MB 2025-02-15 10:23:44,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 377.15 MB 2025-02-15 10:23:44,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 10:23:44,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20164.12 MB 2025-02-15 10:23:44,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1560.28 MB 2025-02-15 10:23:44,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18370.40 MB 2025-02-15 10:23:45,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:23:45,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:23:45,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.06 seconds 2025-02-15 10:23:45,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15657.42 MB 2025-02-15 10:23:45,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15949.38 MB 2025-02-15 10:23:45,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.96 MB 2025-02-15 10:23:45,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20164.12 MB 2025-02-15 10:23:45,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19383.98 MB 2025-02-15 10:23:45,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -780.14 MB 2025-02-15 10:23:45,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19913.04 MB 2025-02-15 10:23:45,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:23:45,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:23:45,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:23:45,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15949.38 MB 2025-02-15 10:23:45,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16989.42 MB 2025-02-15 10:23:45,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1040.04 MB 2025-02-15 10:23:45,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19383.98 MB 2025-02-15 10:23:45,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19904.07 MB 2025-02-15 10:23:45,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 520.09 MB 2025-02-15 10:23:45,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17769.01 MB 2025-02-15 10:23:45,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:23:45,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:23:45,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:23:45,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16989.42 MB 2025-02-15 10:23:45,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18222.47 MB 2025-02-15 10:23:45,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1233.05 MB 2025-02-15 10:23:45,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19904.07 MB 2025-02-15 10:23:45,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:23:45,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3120.56 MB 2025-02-15 10:23:45,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21277.04 MB 2025-02-15 10:23:45,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:23:45,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:23:45,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 10:23:45,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15949.38 MB 2025-02-15 10:23:45,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18222.47 MB 2025-02-15 10:23:45,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2273.09 MB 2025-02-15 10:23:45,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19383.98 MB 2025-02-15 10:23:45,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23024.63 MB 2025-02-15 10:23:45,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3640.66 MB 2025-02-15 10:23:45,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21277.04 MB 2025-02-15 10:23:45,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:23:45,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:23:45,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:23:45,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19065.92 MB 2025-02-15 10:23:45,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19488.82 MB 2025-02-15 10:23:45,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 422.90 MB 2025-02-15 10:23:45,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23024.63 MB 2025-02-15 10:23:45,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-15 10:23:45,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-15 10:23:45,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19878.10 MB 2025-02-15 10:23:45,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:23:45,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:23:45,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:23:45,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19715.91 MB 2025-02-15 10:23:45,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19927.59 MB 2025-02-15 10:23:45,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.67 MB 2025-02-15 10:23:45,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-15 10:23:45,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23253.22 MB 2025-02-15 10:23:45,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 10:23:45,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19976.20 MB 2025-02-15 10:23:45,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:23:45,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:23:45,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.82 seconds 2025-02-15 10:23:45,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:45,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13735.20 MB 2025-02-15 10:23:45,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20128.66 MB 2025-02-15 10:23:45,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6393.46 MB 2025-02-15 10:23:45,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53808.73 MB 2025-02-15 10:23:45,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23253.22 MB 2025-02-15 10:23:45,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30555.50 MB 2025-02-15 10:23:45,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20128.66 MB 2025-02-15 10:23:46,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:23:46,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:23:46,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 10:23:46,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:46,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14876.61 MB 2025-02-15 10:23:46,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17890.64 MB 2025-02-15 10:23:46,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:23:46,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23253.22 MB 2025-02-15 10:23:46,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23253.22 MB 2025-02-15 10:23:46,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:23:46,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18192.01 MB 2025-02-15 10:23:46,240 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:23:46,241 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:23:46,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:23:46,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:23:46,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:23:46,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:23:46,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17890.64 MB 2025-02-15 10:23:46,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26329.67 MB 2025-02-15 10:23:46,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:23:46,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23253.22 MB 2025-02-15 10:23:46,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33743.18 MB 2025-02-15 10:23:46,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:23:46,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26329.67 MB 2025-02-15 10:23:46,517 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:23:46,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:46,519 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:23:46,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:46,521 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:23:46,529 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:23:46,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:23:46,531 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:23:46,532 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:25:18,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:18,845 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:25:18,850 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:25:18,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:18,854 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:25:18,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:18,855 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:25:37,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:25:37,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:25:37,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.41 seconds 2025-02-15 10:25:37,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:37,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21337.47 MB 2025-02-15 10:25:37,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25588.39 MB 2025-02-15 10:25:37,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-15 10:25:37,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46328.18 MB 2025-02-15 10:25:37,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29116.86 MB 2025-02-15 10:25:37,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17211.33 MB 2025-02-15 10:25:37,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34432.72 MB 2025-02-15 10:25:37,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:25:37,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:25:37,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:25:37,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:37,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25588.39 MB 2025-02-15 10:25:37,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22022.52 MB 2025-02-15 10:25:37,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3565.87 MB 2025-02-15 10:25:37,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29116.86 MB 2025-02-15 10:25:37,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39629.88 MB 2025-02-15 10:25:37,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10513.02 MB 2025-02-15 10:25:37,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38200.43 MB 2025-02-15 10:25:39,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:25:39,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:25:39,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 10:25:39,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22022.52 MB 2025-02-15 10:25:39,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22553.37 MB 2025-02-15 10:25:39,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:25:39,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39629.88 MB 2025-02-15 10:25:39,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26990.35 MB 2025-02-15 10:25:39,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12639.54 MB 2025-02-15 10:25:39,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26532.95 MB 2025-02-15 10:25:39,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:25:39,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:25:39,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:25:39,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22553.37 MB 2025-02-15 10:25:39,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24442.90 MB 2025-02-15 10:25:39,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:25:39,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 10:25:39,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27934.06 MB 2025-02-15 10:25:39,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:25:39,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25860.33 MB 2025-02-15 10:25:39,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:25:39,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:25:39,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:25:39,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24442.90 MB 2025-02-15 10:25:39,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26684.76 MB 2025-02-15 10:25:39,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:25:39,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27934.06 MB 2025-02-15 10:25:39,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 10:25:39,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:25:39,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.04 MB 2025-02-15 10:25:39,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:25:39,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:25:39,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:25:39,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22553.37 MB 2025-02-15 10:25:39,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26684.76 MB 2025-02-15 10:25:39,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:25:39,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 10:25:39,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 10:25:39,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:25:39,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.04 MB 2025-02-15 10:25:39,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:25:39,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:25:39,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:25:39,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28218.30 MB 2025-02-15 10:25:39,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28985.30 MB 2025-02-15 10:25:39,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:25:39,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-15 10:25:39,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 10:25:39,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:25:39,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29693.09 MB 2025-02-15 10:25:39,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:25:39,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:25:39,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:25:39,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29398.19 MB 2025-02-15 10:25:39,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29626.87 MB 2025-02-15 10:25:39,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.68 MB 2025-02-15 10:25:39,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 10:25:39,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 10:25:39,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:25:39,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29862.09 MB 2025-02-15 10:25:39,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:25:39,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:25:39,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.86 seconds 2025-02-15 10:25:39,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17153.09 MB 2025-02-15 10:25:39,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29827.80 MB 2025-02-15 10:25:39,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12674.71 MB 2025-02-15 10:25:39,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46328.18 MB 2025-02-15 10:25:39,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 10:25:39,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12314.48 MB 2025-02-15 10:25:39,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29862.09 MB 2025-02-15 10:25:39,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:25:39,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:25:39,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:25:39,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:39,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29827.80 MB 2025-02-15 10:25:39,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22154.86 MB 2025-02-15 10:25:39,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7672.93 MB 2025-02-15 10:25:39,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 10:25:39,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-15 10:25:39,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:25:39,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32337.62 MB 2025-02-15 10:25:40,002 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 10:25:40,002 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:25:40,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:25:40,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:25:40,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:25:40,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:25:40,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22154.86 MB 2025-02-15 10:25:40,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30588.16 MB 2025-02-15 10:25:40,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 10:25:40,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-15 10:25:40,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42398.12 MB 2025-02-15 10:25:40,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 10:25:40,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30588.16 MB 2025-02-15 10:25:40,169 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 10:25:40,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:40,170 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:25:40,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:40,171 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:25:40,176 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:25:40,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:25:40,177 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:25:40,177 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:27:40,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:27:40,725 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:27:40,730 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:27:40,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:27:40,734 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2390, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:27:40,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:27:40,735 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2390, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:28:17,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:28:17,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:28:17,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.86 seconds 2025-02-15 10:28:17,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:17,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29624.44 MB 2025-02-15 10:28:17,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38082.52 MB 2025-02-15 10:28:17,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8458.08 MB 2025-02-15 10:28:17,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59242.45 MB 2025-02-15 10:28:17,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41781.56 MB 2025-02-15 10:28:17,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17460.89 MB 2025-02-15 10:28:17,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47023.05 MB 2025-02-15 10:28:17,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:28:17,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:28:17,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:28:17,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:17,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38082.52 MB 2025-02-15 10:28:17,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28204.68 MB 2025-02-15 10:28:17,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9877.84 MB 2025-02-15 10:28:17,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41781.56 MB 2025-02-15 10:28:17,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59338.92 MB 2025-02-15 10:28:17,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17557.36 MB 2025-02-15 10:28:17,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61291.87 MB 2025-02-15 10:28:19,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:28:19,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:28:19,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:28:19,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:19,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28204.68 MB 2025-02-15 10:28:19,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28735.52 MB 2025-02-15 10:28:19,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:28:19,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59338.92 MB 2025-02-15 10:28:19,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31216.11 MB 2025-02-15 10:28:19,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28122.81 MB 2025-02-15 10:28:19,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32714.07 MB 2025-02-15 10:28:19,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:28:19,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:28:19,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:28:19,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:19,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28735.52 MB 2025-02-15 10:28:19,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30625.06 MB 2025-02-15 10:28:19,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:28:19,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31216.11 MB 2025-02-15 10:28:19,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-15 10:28:19,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:28:19,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32042.49 MB 2025-02-15 10:28:19,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:28:19,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:28:19,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:28:19,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:19,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30625.06 MB 2025-02-15 10:28:19,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32866.91 MB 2025-02-15 10:28:19,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:28:19,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34047.26 MB 2025-02-15 10:28:19,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 10:28:19,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:28:19,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38411.19 MB 2025-02-15 10:28:19,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:28:19,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:28:19,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:28:19,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:19,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28735.52 MB 2025-02-15 10:28:19,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32866.91 MB 2025-02-15 10:28:19,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:28:19,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31216.11 MB 2025-02-15 10:28:19,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 10:28:19,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:28:19,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38411.19 MB 2025-02-15 10:28:20,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:28:20,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:28:20,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:28:20,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:20,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34400.46 MB 2025-02-15 10:28:20,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35167.46 MB 2025-02-15 10:28:20,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:28:20,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40181.43 MB 2025-02-15 10:28:20,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40598.77 MB 2025-02-15 10:28:20,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:28:20,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35875.25 MB 2025-02-15 10:28:20,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:28:20,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:28:20,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:28:20,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:20,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35580.35 MB 2025-02-15 10:28:20,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35809.54 MB 2025-02-15 10:28:20,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.19 MB 2025-02-15 10:28:20,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40598.77 MB 2025-02-15 10:28:20,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40598.77 MB 2025-02-15 10:28:20,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:28:20,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36009.30 MB 2025-02-15 10:28:20,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:28:20,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:28:20,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.45 seconds 2025-02-15 10:28:20,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:20,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21296.57 MB 2025-02-15 10:28:20,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36010.61 MB 2025-02-15 10:28:20,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14714.04 MB 2025-02-15 10:28:20,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55012.49 MB 2025-02-15 10:28:20,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40598.77 MB 2025-02-15 10:28:20,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14413.73 MB 2025-02-15 10:28:20,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36010.61 MB 2025-02-15 10:28:20,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:28:20,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:28:20,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:28:20,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:20,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36010.61 MB 2025-02-15 10:28:20,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26300.96 MB 2025-02-15 10:28:20,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9709.65 MB 2025-02-15 10:28:20,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40598.77 MB 2025-02-15 10:28:20,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40598.77 MB 2025-02-15 10:28:20,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:28:20,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38522.28 MB 2025-02-15 10:28:20,474 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:28:20,475 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:28:20,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:28:20,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:28:20,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:28:20,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:28:20,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26300.96 MB 2025-02-15 10:28:20,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34739.99 MB 2025-02-15 10:28:20,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:28:20,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40598.77 MB 2025-02-15 10:28:20,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48989.47 MB 2025-02-15 10:28:20,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:28:20,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34739.99 MB 2025-02-15 10:28:20,647 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:28:20,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:28:20,648 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:28:20,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:28:20,649 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:28:20,654 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:28:20,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:28:20,655 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:28:20,655 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:29:12,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:12,566 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:29:12,571 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:29:12,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:12,575 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2518, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:29:12,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:12,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2518, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:29:51,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:29:51,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:29:51,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.12 seconds 2025-02-15 10:29:51,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:51,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30517.67 MB 2025-02-15 10:29:51,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39428.73 MB 2025-02-15 10:29:51,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8911.06 MB 2025-02-15 10:29:51,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79123.45 MB 2025-02-15 10:29:51,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42939.19 MB 2025-02-15 10:29:51,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36184.26 MB 2025-02-15 10:29:51,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48369.26 MB 2025-02-15 10:29:51,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:29:51,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:29:51,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 10:29:51,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:51,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39428.73 MB 2025-02-15 10:29:51,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28871.42 MB 2025-02-15 10:29:51,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10557.32 MB 2025-02-15 10:29:51,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42939.19 MB 2025-02-15 10:29:51,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62258.15 MB 2025-02-15 10:29:51,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19318.96 MB 2025-02-15 10:29:51,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65286.66 MB 2025-02-15 10:29:53,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:29:53,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:29:53,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 10:29:53,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:53,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28871.42 MB 2025-02-15 10:29:53,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29402.26 MB 2025-02-15 10:29:53,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:29:53,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62258.15 MB 2025-02-15 10:29:53,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31694.26 MB 2025-02-15 10:29:53,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30563.89 MB 2025-02-15 10:29:53,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33380.81 MB 2025-02-15 10:29:53,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:29:53,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:29:53,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:29:53,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:53,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29402.26 MB 2025-02-15 10:29:53,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31291.79 MB 2025-02-15 10:29:53,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:29:53,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31694.26 MB 2025-02-15 10:29:53,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34525.41 MB 2025-02-15 10:29:53,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:29:53,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32709.22 MB 2025-02-15 10:29:54,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:29:54,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:29:54,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:29:54,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31291.79 MB 2025-02-15 10:29:54,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33533.65 MB 2025-02-15 10:29:54,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:29:54,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34525.41 MB 2025-02-15 10:29:54,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40659.58 MB 2025-02-15 10:29:54,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:29:54,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39077.93 MB 2025-02-15 10:29:54,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:29:54,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:29:54,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:29:54,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29402.26 MB 2025-02-15 10:29:54,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33533.65 MB 2025-02-15 10:29:54,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:29:54,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31694.26 MB 2025-02-15 10:29:54,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40659.58 MB 2025-02-15 10:29:54,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 10:29:54,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39077.93 MB 2025-02-15 10:29:54,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:29:54,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:29:54,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:29:54,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35067.19 MB 2025-02-15 10:29:54,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35834.19 MB 2025-02-15 10:29:54,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:29:54,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40659.58 MB 2025-02-15 10:29:54,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41076.92 MB 2025-02-15 10:29:54,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:29:54,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36541.98 MB 2025-02-15 10:29:54,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:29:54,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:29:54,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:29:54,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36247.08 MB 2025-02-15 10:29:54,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36475.18 MB 2025-02-15 10:29:54,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-15 10:29:54,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41076.92 MB 2025-02-15 10:29:54,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41076.92 MB 2025-02-15 10:29:54,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:29:54,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36690.40 MB 2025-02-15 10:29:54,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:29:54,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:29:54,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.74 seconds 2025-02-15 10:29:54,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21743.19 MB 2025-02-15 10:29:54,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36675.20 MB 2025-02-15 10:29:54,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14932.01 MB 2025-02-15 10:29:54,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70348.96 MB 2025-02-15 10:29:54,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41076.92 MB 2025-02-15 10:29:54,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29272.05 MB 2025-02-15 10:29:54,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36690.40 MB 2025-02-15 10:29:54,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:29:54,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:29:54,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:29:54,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36675.20 MB 2025-02-15 10:29:54,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26732.04 MB 2025-02-15 10:29:54,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9943.16 MB 2025-02-15 10:29:54,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41076.92 MB 2025-02-15 10:29:54,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41076.92 MB 2025-02-15 10:29:54,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:29:54,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39173.66 MB 2025-02-15 10:29:54,610 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 10:29:54,610 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:29:54,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:29:54,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:29:54,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:29:54,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:29:54,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26732.04 MB 2025-02-15 10:29:54,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35127.25 MB 2025-02-15 10:29:54,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-15 10:29:54,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41076.92 MB 2025-02-15 10:29:54,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45250.25 MB 2025-02-15 10:29:54,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 10:29:54,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35127.25 MB 2025-02-15 10:29:54,775 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 10:29:54,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:54,777 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:29:54,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:54,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:29:54,783 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:29:54,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:29:54,784 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:29:54,784 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:30:44,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:30:44,474 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:30:44,482 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:30:44,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:30:44,488 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1053, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:30:44,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:30:44,490 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1053, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:31:00,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:31:00,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:31:00,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.38 seconds 2025-02-15 10:31:00,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:00,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20306.18 MB 2025-02-15 10:31:00,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24032.82 MB 2025-02-15 10:31:00,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3726.64 MB 2025-02-15 10:31:00,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53596.91 MB 2025-02-15 10:31:00,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28695.33 MB 2025-02-15 10:31:00,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24901.58 MB 2025-02-15 10:31:00,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32948.44 MB 2025-02-15 10:31:00,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:31:00,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:31:00,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:31:00,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:00,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24032.82 MB 2025-02-15 10:31:00,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21253.12 MB 2025-02-15 10:31:00,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2779.70 MB 2025-02-15 10:31:00,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28695.33 MB 2025-02-15 10:31:00,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38140.90 MB 2025-02-15 10:31:00,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9445.57 MB 2025-02-15 10:31:00,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35576.80 MB 2025-02-15 10:31:02,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:31:02,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:31:02,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:31:02,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:02,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21253.12 MB 2025-02-15 10:31:02,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21783.96 MB 2025-02-15 10:31:02,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:31:02,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38140.90 MB 2025-02-15 10:31:02,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27093.11 MB 2025-02-15 10:31:02,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11047.80 MB 2025-02-15 10:31:02,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25762.51 MB 2025-02-15 10:31:02,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:31:02,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:31:02,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:31:02,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:02,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.96 MB 2025-02-15 10:31:02,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23673.49 MB 2025-02-15 10:31:02,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:31:02,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27093.11 MB 2025-02-15 10:31:02,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28036.83 MB 2025-02-15 10:31:02,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:31:02,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25090.92 MB 2025-02-15 10:31:03,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:31:03,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:31:03,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:31:03,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23673.49 MB 2025-02-15 10:31:03,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25915.35 MB 2025-02-15 10:31:03,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:31:03,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28036.83 MB 2025-02-15 10:31:03,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33699.14 MB 2025-02-15 10:31:03,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:31:03,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31459.63 MB 2025-02-15 10:31:03,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:31:03,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:31:03,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:31:03,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.96 MB 2025-02-15 10:31:03,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25915.35 MB 2025-02-15 10:31:03,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:31:03,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27093.11 MB 2025-02-15 10:31:03,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33699.14 MB 2025-02-15 10:31:03,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:31:03,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31459.63 MB 2025-02-15 10:31:03,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:31:03,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:31:03,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:31:03,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27448.89 MB 2025-02-15 10:31:03,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28215.89 MB 2025-02-15 10:31:03,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:31:03,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33699.14 MB 2025-02-15 10:31:03,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34116.47 MB 2025-02-15 10:31:03,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:31:03,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28923.68 MB 2025-02-15 10:31:03,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:31:03,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:31:03,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:31:03,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28628.78 MB 2025-02-15 10:31:03,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28856.67 MB 2025-02-15 10:31:03,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.89 MB 2025-02-15 10:31:03,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34116.47 MB 2025-02-15 10:31:03,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34116.47 MB 2025-02-15 10:31:03,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:31:03,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29100.97 MB 2025-02-15 10:31:03,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:31:03,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:31:03,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.81 seconds 2025-02-15 10:31:03,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16637.44 MB 2025-02-15 10:31:03,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29057.67 MB 2025-02-15 10:31:03,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12420.23 MB 2025-02-15 10:31:03,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53596.91 MB 2025-02-15 10:31:03,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34116.47 MB 2025-02-15 10:31:03,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19480.44 MB 2025-02-15 10:31:03,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29100.97 MB 2025-02-15 10:31:03,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:31:03,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:31:03,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:31:03,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29057.67 MB 2025-02-15 10:31:03,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21640.69 MB 2025-02-15 10:31:03,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7416.98 MB 2025-02-15 10:31:03,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34116.47 MB 2025-02-15 10:31:03,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34116.47 MB 2025-02-15 10:31:03,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:31:03,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.42 MB 2025-02-15 10:31:03,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 10:31:03,591 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:31:03,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:31:03,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:31:03,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:31:03,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:03,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.69 MB 2025-02-15 10:31:03,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30076.28 MB 2025-02-15 10:31:03,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 10:31:03,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34116.47 MB 2025-02-15 10:31:03,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42505.08 MB 2025-02-15 10:31:03,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 10:31:03,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30076.28 MB 2025-02-15 10:31:03,760 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 10:31:03,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:03,761 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:31:03,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:03,762 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:31:03,767 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:31:03,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:03,768 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:31:03,768 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:31:13,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:13,015 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:31:13,020 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:31:13,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:13,024 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1098, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:31:13,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:13,025 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1098, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:31:30,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:31:30,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:31:30,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.16 seconds 2025-02-15 10:31:30,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:30,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20619.75 MB 2025-02-15 10:31:30,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.77 MB 2025-02-15 10:31:30,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3886.02 MB 2025-02-15 10:31:30,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50893.68 MB 2025-02-15 10:31:30,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28733.08 MB 2025-02-15 10:31:30,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22160.61 MB 2025-02-15 10:31:30,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33488.50 MB 2025-02-15 10:31:30,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:31:30,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:31:30,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:31:30,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:30,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24505.77 MB 2025-02-15 10:31:30,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21487.06 MB 2025-02-15 10:31:30,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3018.71 MB 2025-02-15 10:31:30,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28733.08 MB 2025-02-15 10:31:30,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38516.29 MB 2025-02-15 10:31:30,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9783.21 MB 2025-02-15 10:31:30,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36387.35 MB 2025-02-15 10:31:32,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:31:32,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:31:32,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:31:32,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21487.06 MB 2025-02-15 10:31:32,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22017.90 MB 2025-02-15 10:31:32,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:31:32,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38516.29 MB 2025-02-15 10:31:32,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26971.47 MB 2025-02-15 10:31:32,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11544.82 MB 2025-02-15 10:31:32,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25996.45 MB 2025-02-15 10:31:32,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:31:32,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:31:32,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:31:32,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22017.90 MB 2025-02-15 10:31:32,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23907.43 MB 2025-02-15 10:31:32,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:31:32,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-15 10:31:32,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27915.19 MB 2025-02-15 10:31:32,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:31:32,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25324.86 MB 2025-02-15 10:31:32,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:31:32,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:31:32,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:31:32,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23907.43 MB 2025-02-15 10:31:32,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26149.29 MB 2025-02-15 10:31:32,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:31:32,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27915.19 MB 2025-02-15 10:31:32,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-15 10:31:32,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:31:32,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31693.57 MB 2025-02-15 10:31:32,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:31:32,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:31:32,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:31:32,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22017.90 MB 2025-02-15 10:31:32,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26149.29 MB 2025-02-15 10:31:32,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:31:32,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-15 10:31:32,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-15 10:31:32,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:31:32,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31693.57 MB 2025-02-15 10:31:32,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:31:32,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:31:32,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 10:31:32,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27682.83 MB 2025-02-15 10:31:32,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28449.83 MB 2025-02-15 10:31:32,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:31:32,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33577.50 MB 2025-02-15 10:31:32,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33992.74 MB 2025-02-15 10:31:32,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:31:32,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29157.62 MB 2025-02-15 10:31:32,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:31:32,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:31:32,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:31:32,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28862.72 MB 2025-02-15 10:31:32,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29089.87 MB 2025-02-15 10:31:32,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.14 MB 2025-02-15 10:31:32,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33992.74 MB 2025-02-15 10:31:32,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33992.74 MB 2025-02-15 10:31:32,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:31:32,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29321.41 MB 2025-02-15 10:31:32,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:31:32,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:31:32,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.65 seconds 2025-02-15 10:31:32,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16794.23 MB 2025-02-15 10:31:32,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.72 MB 2025-02-15 10:31:32,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12496.49 MB 2025-02-15 10:31:32,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50893.68 MB 2025-02-15 10:31:32,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33992.74 MB 2025-02-15 10:31:32,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16900.95 MB 2025-02-15 10:31:32,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29321.41 MB 2025-02-15 10:31:32,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:31:32,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:31:32,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:31:32,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:32,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29290.72 MB 2025-02-15 10:31:32,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21789.49 MB 2025-02-15 10:31:32,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7501.23 MB 2025-02-15 10:31:32,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33992.74 MB 2025-02-15 10:31:32,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33992.74 MB 2025-02-15 10:31:32,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:31:32,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31794.70 MB 2025-02-15 10:31:32,993 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 10:31:32,994 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:31:33,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:31:33,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:31:33,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:31:33,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:31:33,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21789.49 MB 2025-02-15 10:31:33,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30203.01 MB 2025-02-15 10:31:33,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-15 10:31:33,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33992.74 MB 2025-02-15 10:31:33,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42356.18 MB 2025-02-15 10:31:33,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 10:31:33,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30203.01 MB 2025-02-15 10:31:33,252 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 10:31:33,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:33,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:31:33,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:33,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:31:33,264 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:31:33,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:31:33,266 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:31:33,266 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:32:34,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:34,090 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:32:34,095 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:32:34,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:34,100 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:32:34,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:34,101 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:32:36,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:32:36,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:32:36,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-15 10:32:36,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:36,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-15 10:32:36,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-15 10:32:36,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-15 10:32:36,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50719.62 MB 2025-02-15 10:32:36,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19566.43 MB 2025-02-15 10:32:36,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31153.19 MB 2025-02-15 10:32:36,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23722.22 MB 2025-02-15 10:32:36,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:32:36,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:32:36,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:32:36,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:36,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-15 10:32:36,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15147.27 MB 2025-02-15 10:32:36,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.26 MB 2025-02-15 10:32:36,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19566.43 MB 2025-02-15 10:32:36,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19566.43 MB 2025-02-15 10:32:36,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:32:36,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.09 MB 2025-02-15 10:32:37,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:32:37,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:32:37,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-15 10:32:37,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15147.27 MB 2025-02-15 10:32:37,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15378.19 MB 2025-02-15 10:32:37,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.92 MB 2025-02-15 10:32:37,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19566.43 MB 2025-02-15 10:32:37,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19566.43 MB 2025-02-15 10:32:37,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:32:37,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19317.96 MB 2025-02-15 10:32:37,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:32:37,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:32:37,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:32:37,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.12 MB 2025-02-15 10:32:37,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16199.87 MB 2025-02-15 10:32:37,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 821.75 MB 2025-02-15 10:32:37,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19566.43 MB 2025-02-15 10:32:37,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19566.43 MB 2025-02-15 10:32:37,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:32:37,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16816.46 MB 2025-02-15 10:32:37,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:32:37,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:32:37,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:32:37,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16199.87 MB 2025-02-15 10:32:37,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17175.11 MB 2025-02-15 10:32:37,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 975.24 MB 2025-02-15 10:32:37,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19566.43 MB 2025-02-15 10:32:37,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21416.12 MB 2025-02-15 10:32:37,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1849.69 MB 2025-02-15 10:32:37,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19589.20 MB 2025-02-15 10:32:37,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:32:37,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:32:37,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:32:37,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.12 MB 2025-02-15 10:32:37,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17175.11 MB 2025-02-15 10:32:37,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1796.99 MB 2025-02-15 10:32:37,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19566.43 MB 2025-02-15 10:32:37,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21416.12 MB 2025-02-15 10:32:37,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1849.69 MB 2025-02-15 10:32:37,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19589.20 MB 2025-02-15 10:32:37,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:32:37,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:32:37,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:32:37,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17842.20 MB 2025-02-15 10:32:37,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18176.11 MB 2025-02-15 10:32:37,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 333.91 MB 2025-02-15 10:32:37,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21416.12 MB 2025-02-15 10:32:37,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21594.37 MB 2025-02-15 10:32:37,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-15 10:32:37,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18490.00 MB 2025-02-15 10:32:37,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:32:37,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:32:37,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:32:37,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18355.73 MB 2025-02-15 10:32:37,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18574.45 MB 2025-02-15 10:32:37,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.72 MB 2025-02-15 10:32:37,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21594.37 MB 2025-02-15 10:32:37,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21594.37 MB 2025-02-15 10:32:37,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:32:37,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18599.77 MB 2025-02-15 10:32:37,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:32:37,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:32:37,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.89 seconds 2025-02-15 10:32:37,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:37,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-15 10:32:37,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18775.30 MB 2025-02-15 10:32:37,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5165.52 MB 2025-02-15 10:32:37,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50719.62 MB 2025-02-15 10:32:37,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21594.37 MB 2025-02-15 10:32:37,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29125.25 MB 2025-02-15 10:32:37,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18775.30 MB 2025-02-15 10:32:38,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:32:38,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:32:38,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:32:38,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:38,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18775.30 MB 2025-02-15 10:32:38,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17541.62 MB 2025-02-15 10:32:38,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1233.68 MB 2025-02-15 10:32:38,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21594.37 MB 2025-02-15 10:32:38,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21594.37 MB 2025-02-15 10:32:38,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:32:38,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19009.95 MB 2025-02-15 10:32:38,278 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 10:32:38,279 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:32:38,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:32:38,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:32:38,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:32:38,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:32:38,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17541.62 MB 2025-02-15 10:32:38,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25963.94 MB 2025-02-15 10:32:38,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 10:32:38,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21594.37 MB 2025-02-15 10:32:38,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32063.36 MB 2025-02-15 10:32:38,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 10:32:38,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25963.94 MB 2025-02-15 10:32:38,529 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 10:32:38,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:38,531 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:32:38,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:38,533 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:32:38,541 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:32:38,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:32:38,543 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:32:38,543 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:34:22,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:22,889 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:34:22,894 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:34:22,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:22,898 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1579, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:34:22,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:22,899 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1579, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:34:47,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:34:47,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:34:47,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.24 seconds 2025-02-15 10:34:47,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:47,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23971.43 MB 2025-02-15 10:34:47,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29560.34 MB 2025-02-15 10:34:47,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5588.91 MB 2025-02-15 10:34:47,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44623.20 MB 2025-02-15 10:34:47,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38811.99 MB 2025-02-15 10:34:47,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5811.21 MB 2025-02-15 10:34:47,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38425.64 MB 2025-02-15 10:34:47,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:34:47,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:34:47,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:34:47,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:47,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29560.34 MB 2025-02-15 10:34:47,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23986.58 MB 2025-02-15 10:34:47,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5573.76 MB 2025-02-15 10:34:47,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38811.99 MB 2025-02-15 10:34:47,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42773.51 MB 2025-02-15 10:34:47,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3961.52 MB 2025-02-15 10:34:47,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38834.68 MB 2025-02-15 10:34:49,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:34:49,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:34:49,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:34:49,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23986.58 MB 2025-02-15 10:34:49,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24517.42 MB 2025-02-15 10:34:49,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:34:49,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42773.51 MB 2025-02-15 10:34:49,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30452.74 MB 2025-02-15 10:34:49,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12320.77 MB 2025-02-15 10:34:49,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28495.97 MB 2025-02-15 10:34:49,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:34:49,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:34:49,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:34:49,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24517.42 MB 2025-02-15 10:34:49,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26406.96 MB 2025-02-15 10:34:49,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:34:49,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30452.74 MB 2025-02-15 10:34:49,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30452.74 MB 2025-02-15 10:34:49,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:34:49,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27824.38 MB 2025-02-15 10:34:49,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:34:49,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:34:49,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 10:34:49,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26406.96 MB 2025-02-15 10:34:49,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28648.81 MB 2025-02-15 10:34:49,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:34:49,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30452.74 MB 2025-02-15 10:34:49,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36115.05 MB 2025-02-15 10:34:49,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:34:49,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34193.09 MB 2025-02-15 10:34:49,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:34:49,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:34:49,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 10:34:49,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24517.42 MB 2025-02-15 10:34:49,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28648.81 MB 2025-02-15 10:34:49,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:34:49,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30452.74 MB 2025-02-15 10:34:49,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36115.05 MB 2025-02-15 10:34:49,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:34:49,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34193.09 MB 2025-02-15 10:34:49,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:34:49,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:34:49,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:34:49,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30182.35 MB 2025-02-15 10:34:49,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30949.36 MB 2025-02-15 10:34:49,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:34:49,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36115.05 MB 2025-02-15 10:34:49,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36532.39 MB 2025-02-15 10:34:49,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:34:49,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31657.14 MB 2025-02-15 10:34:49,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:34:49,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:34:49,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:34:49,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31362.24 MB 2025-02-15 10:34:49,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31591.28 MB 2025-02-15 10:34:49,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-15 10:34:49,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36532.39 MB 2025-02-15 10:34:49,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36532.39 MB 2025-02-15 10:34:49,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:34:49,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31778.33 MB 2025-02-15 10:34:49,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:34:49,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:34:49,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.72 seconds 2025-02-15 10:34:49,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18470.07 MB 2025-02-15 10:34:49,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31792.35 MB 2025-02-15 10:34:49,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13322.28 MB 2025-02-15 10:34:49,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44623.20 MB 2025-02-15 10:34:49,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36532.39 MB 2025-02-15 10:34:49,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8090.81 MB 2025-02-15 10:34:49,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.35 MB 2025-02-15 10:34:49,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:34:49,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:34:49,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:34:49,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31792.35 MB 2025-02-15 10:34:49,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23474.46 MB 2025-02-15 10:34:49,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8317.89 MB 2025-02-15 10:34:49,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36532.39 MB 2025-02-15 10:34:49,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36532.39 MB 2025-02-15 10:34:49,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:34:49,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34304.02 MB 2025-02-15 10:34:49,905 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:34:49,905 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:34:49,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:34:49,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:34:49,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:34:49,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:34:49,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23474.46 MB 2025-02-15 10:34:49,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31913.48 MB 2025-02-15 10:34:49,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:34:49,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36532.39 MB 2025-02-15 10:34:49,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44923.09 MB 2025-02-15 10:34:49,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:34:49,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31913.48 MB 2025-02-15 10:34:50,070 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:34:50,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:50,071 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:34:50,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:50,072 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:34:50,077 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:34:50,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:50,078 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:34:50,078 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:34:57,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:57,668 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:34:57,676 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:34:57,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:57,682 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:34:57,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:34:57,684 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:35:31,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:35:31,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:35:31,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.63 seconds 2025-02-15 10:35:31,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:31,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27978.12 MB 2025-02-15 10:35:31,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35601.27 MB 2025-02-15 10:35:31,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7623.15 MB 2025-02-15 10:35:31,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57508.10 MB 2025-02-15 10:35:31,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40856.72 MB 2025-02-15 10:35:31,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16651.39 MB 2025-02-15 10:35:31,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44470.76 MB 2025-02-15 10:35:31,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:35:31,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:35:31,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.33 seconds 2025-02-15 10:35:31,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:31,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35601.27 MB 2025-02-15 10:35:31,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26976.87 MB 2025-02-15 10:35:31,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8624.40 MB 2025-02-15 10:35:31,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40856.72 MB 2025-02-15 10:35:31,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57331.94 MB 2025-02-15 10:35:31,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16475.23 MB 2025-02-15 10:35:31,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57183.73 MB 2025-02-15 10:35:33,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:35:33,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:35:33,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 10:35:33,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:33,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26976.87 MB 2025-02-15 10:35:33,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27507.72 MB 2025-02-15 10:35:33,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:35:33,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57331.94 MB 2025-02-15 10:35:33,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31172.07 MB 2025-02-15 10:35:33,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26159.87 MB 2025-02-15 10:35:33,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31487.30 MB 2025-02-15 10:35:33,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:35:33,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:35:33,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:35:33,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:33,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-15 10:35:33,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29397.25 MB 2025-02-15 10:35:33,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:35:33,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31172.07 MB 2025-02-15 10:35:33,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33059.50 MB 2025-02-15 10:35:33,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:35:33,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30814.68 MB 2025-02-15 10:35:33,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:35:33,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:35:33,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:35:33,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:33,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29397.25 MB 2025-02-15 10:35:33,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31639.11 MB 2025-02-15 10:35:33,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:35:33,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33059.50 MB 2025-02-15 10:35:33,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38721.81 MB 2025-02-15 10:35:33,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:35:33,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.39 MB 2025-02-15 10:35:33,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:35:33,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:35:33,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:35:33,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:33,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-15 10:35:33,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31639.11 MB 2025-02-15 10:35:33,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:35:33,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31172.07 MB 2025-02-15 10:35:33,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38721.81 MB 2025-02-15 10:35:33,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 10:35:33,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.39 MB 2025-02-15 10:35:34,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:35:34,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:35:34,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:35:34,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:34,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33172.65 MB 2025-02-15 10:35:34,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33939.65 MB 2025-02-15 10:35:34,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:35:34,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38721.81 MB 2025-02-15 10:35:34,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 10:35:34,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:35:34,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34647.44 MB 2025-02-15 10:35:34,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:35:34,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:35:34,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:35:34,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:34,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34352.54 MB 2025-02-15 10:35:34,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34581.75 MB 2025-02-15 10:35:34,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-15 10:35:34,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 10:35:34,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 10:35:34,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:34,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34812.20 MB 2025-02-15 10:35:34,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:35:34,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:35:34,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.34 seconds 2025-02-15 10:35:34,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:34,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-15 10:35:34,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34782.82 MB 2025-02-15 10:35:34,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14309.40 MB 2025-02-15 10:35:34,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57508.10 MB 2025-02-15 10:35:34,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 10:35:34,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18371.05 MB 2025-02-15 10:35:34,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34812.20 MB 2025-02-15 10:35:34,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:35:34,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:35:34,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:35:34,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:34,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22463.77 MB 2025-02-15 10:35:34,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25477.80 MB 2025-02-15 10:35:34,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:35:34,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 10:35:34,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 10:35:34,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:34,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25779.17 MB 2025-02-15 10:35:34,312 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:35:34,313 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:35:34,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:35:34,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:35:34,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:35:34,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:34,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25477.80 MB 2025-02-15 10:35:34,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33916.83 MB 2025-02-15 10:35:34,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:35:34,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 10:35:34,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47527.76 MB 2025-02-15 10:35:34,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:35:34,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.83 MB 2025-02-15 10:35:34,478 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:35:34,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:34,479 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:35:34,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:34,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:35:34,485 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:35:34,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:34,486 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:35:34,486 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:35:56,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:56,325 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:35:56,330 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:35:56,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:56,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:35:56,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:56,334 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:35:57,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:35:57,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:35:57,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.37 seconds 2025-02-15 10:35:57,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:57,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-15 10:35:57,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13872.32 MB 2025-02-15 10:35:57,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-15 10:35:57,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60112.76 MB 2025-02-15 10:35:57,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17186.16 MB 2025-02-15 10:35:57,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42926.60 MB 2025-02-15 10:35:57,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-15 10:35:57,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:35:57,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:35:57,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:35:57,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:57,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13872.32 MB 2025-02-15 10:35:57,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14019.77 MB 2025-02-15 10:35:57,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-15 10:35:57,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17186.16 MB 2025-02-15 10:35:57,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17186.16 MB 2025-02-15 10:35:57,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:57,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14476.36 MB 2025-02-15 10:35:58,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:35:58,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:35:58,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-15 10:35:58,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14019.77 MB 2025-02-15 10:35:58,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14133.84 MB 2025-02-15 10:35:58,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.07 MB 2025-02-15 10:35:58,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17186.16 MB 2025-02-15 10:35:58,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17588.81 MB 2025-02-15 10:35:58,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 10:35:58,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18105.46 MB 2025-02-15 10:35:58,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:35:58,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:35:58,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:35:58,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-15 10:35:58,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14539.99 MB 2025-02-15 10:35:58,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-15 10:35:58,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-15 10:35:58,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17588.81 MB 2025-02-15 10:35:58,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:58,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.74 MB 2025-02-15 10:35:58,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:35:58,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:35:58,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:35:58,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14539.99 MB 2025-02-15 10:35:58,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-15 10:35:58,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-15 10:35:58,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-15 10:35:58,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17588.81 MB 2025-02-15 10:35:58,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:58,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-15 10:35:58,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:35:58,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:35:58,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:35:58,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-15 10:35:58,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-15 10:35:58,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-15 10:35:58,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-15 10:35:58,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17588.81 MB 2025-02-15 10:35:58,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:58,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-15 10:35:58,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:35:58,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:35:58,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:35:58,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15510.28 MB 2025-02-15 10:35:58,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15717.46 MB 2025-02-15 10:35:58,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.18 MB 2025-02-15 10:35:58,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-15 10:35:58,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17720.93 MB 2025-02-15 10:35:58,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-15 10:35:58,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15869.63 MB 2025-02-15 10:35:58,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:35:58,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:35:58,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:35:58,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15848.51 MB 2025-02-15 10:35:58,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16054.36 MB 2025-02-15 10:35:58,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.85 MB 2025-02-15 10:35:58,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17720.93 MB 2025-02-15 10:35:58,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17720.93 MB 2025-02-15 10:35:58,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:35:58,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16054.36 MB 2025-02-15 10:35:58,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:35:58,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:35:58,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 10:35:58,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13268.34 MB 2025-02-15 10:35:58,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16237.85 MB 2025-02-15 10:35:58,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2969.52 MB 2025-02-15 10:35:58,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60112.76 MB 2025-02-15 10:35:58,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17720.93 MB 2025-02-15 10:35:58,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42391.83 MB 2025-02-15 10:35:58,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16237.85 MB 2025-02-15 10:35:58,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:35:58,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:35:58,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 10:35:58,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16237.85 MB 2025-02-15 10:35:58,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18988.31 MB 2025-02-15 10:35:58,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-15 10:35:58,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17720.93 MB 2025-02-15 10:35:58,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20319.31 MB 2025-02-15 10:35:58,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2598.37 MB 2025-02-15 10:35:58,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19263.32 MB 2025-02-15 10:35:58,545 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-15 10:35:58,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:35:58,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:35:58,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:35:58,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:35:58,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:35:58,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18988.31 MB 2025-02-15 10:35:58,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26689.12 MB 2025-02-15 10:35:58,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-15 10:35:58,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20319.31 MB 2025-02-15 10:35:58,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29890.71 MB 2025-02-15 10:35:58,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9571.40 MB 2025-02-15 10:35:58,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26689.12 MB 2025-02-15 10:35:58,696 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-15 10:35:58,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:58,698 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:35:58,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:58,699 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:35:58,704 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:35:58,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:35:58,705 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:35:58,705 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:37:08,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:08,953 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:37:08,958 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:37:08,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:08,963 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 492, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:37:08,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:08,964 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 492, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:37:16,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:37:16,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:37:16,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.59 seconds 2025-02-15 10:37:16,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:16,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16397.43 MB 2025-02-15 10:37:16,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18138.59 MB 2025-02-15 10:37:16,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1741.16 MB 2025-02-15 10:37:16,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41376.81 MB 2025-02-15 10:37:16,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22223.52 MB 2025-02-15 10:37:16,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19153.29 MB 2025-02-15 10:37:16,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27001.26 MB 2025-02-15 10:37:16,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:37:16,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:37:16,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 10:37:16,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:16,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18138.59 MB 2025-02-15 10:37:16,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.99 MB 2025-02-15 10:37:16,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.40 MB 2025-02-15 10:37:16,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22223.52 MB 2025-02-15 10:37:16,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26935.82 MB 2025-02-15 10:37:16,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4712.30 MB 2025-02-15 10:37:16,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25657.76 MB 2025-02-15 10:37:18,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:37:18,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:37:18,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 10:37:18,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18335.99 MB 2025-02-15 10:37:18,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18866.83 MB 2025-02-15 10:37:18,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:37:18,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26935.82 MB 2025-02-15 10:37:18,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23639.10 MB 2025-02-15 10:37:18,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3296.72 MB 2025-02-15 10:37:18,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22845.79 MB 2025-02-15 10:37:18,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:37:18,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:37:18,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:37:18,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18866.83 MB 2025-02-15 10:37:18,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20756.37 MB 2025-02-15 10:37:18,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:37:18,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23639.10 MB 2025-02-15 10:37:18,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25526.53 MB 2025-02-15 10:37:18,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:37:18,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22173.80 MB 2025-02-15 10:37:18,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:37:18,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:37:18,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:37:18,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20756.37 MB 2025-02-15 10:37:18,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22998.22 MB 2025-02-15 10:37:18,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:37:18,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25526.53 MB 2025-02-15 10:37:18,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30716.99 MB 2025-02-15 10:37:18,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 10:37:18,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28542.51 MB 2025-02-15 10:37:18,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:37:18,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:37:18,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:37:18,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18866.83 MB 2025-02-15 10:37:18,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22998.22 MB 2025-02-15 10:37:18,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:37:18,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23639.10 MB 2025-02-15 10:37:18,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30716.99 MB 2025-02-15 10:37:18,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 10:37:18,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28542.51 MB 2025-02-15 10:37:18,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:37:18,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:37:18,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:37:18,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24531.77 MB 2025-02-15 10:37:18,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25298.77 MB 2025-02-15 10:37:18,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:37:18,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30716.99 MB 2025-02-15 10:37:18,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31132.22 MB 2025-02-15 10:37:18,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:37:18,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26006.56 MB 2025-02-15 10:37:18,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:37:18,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:37:18,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:37:18,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25711.66 MB 2025-02-15 10:37:18,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25940.67 MB 2025-02-15 10:37:18,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-15 10:37:18,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31132.22 MB 2025-02-15 10:37:18,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31132.22 MB 2025-02-15 10:37:18,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:37:18,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26133.55 MB 2025-02-15 10:37:18,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:37:18,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:37:18,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.95 seconds 2025-02-15 10:37:18,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:18,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14683.26 MB 2025-02-15 10:37:18,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26141.74 MB 2025-02-15 10:37:18,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11458.48 MB 2025-02-15 10:37:18,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41376.81 MB 2025-02-15 10:37:18,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31132.22 MB 2025-02-15 10:37:18,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10244.59 MB 2025-02-15 10:37:18,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26141.74 MB 2025-02-15 10:37:19,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:37:19,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:37:19,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:37:19,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:19,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26141.74 MB 2025-02-15 10:37:19,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.65 MB 2025-02-15 10:37:19,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6454.09 MB 2025-02-15 10:37:19,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31132.22 MB 2025-02-15 10:37:19,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31132.22 MB 2025-02-15 10:37:19,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:37:19,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28653.41 MB 2025-02-15 10:37:19,202 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:37:19,202 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:37:19,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:37:19,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:37:19,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:37:19,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:37:19,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.65 MB 2025-02-15 10:37:19,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28126.67 MB 2025-02-15 10:37:19,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:37:19,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31132.22 MB 2025-02-15 10:37:19,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41622.18 MB 2025-02-15 10:37:19,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:37:19,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28126.67 MB 2025-02-15 10:37:19,370 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:37:19,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:19,372 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:37:19,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:19,373 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:37:19,377 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:37:19,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:19,378 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:37:19,379 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:37:48,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:48,232 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:37:48,239 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:37:48,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:48,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1581, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:37:48,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:37:48,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1581, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:38:12,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:38:12,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:38:12,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.63 seconds 2025-02-15 10:38:12,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:12,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23985.37 MB 2025-02-15 10:38:12,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29580.57 MB 2025-02-15 10:38:12,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5595.20 MB 2025-02-15 10:38:12,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54207.18 MB 2025-02-15 10:38:12,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38486.93 MB 2025-02-15 10:38:12,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15720.25 MB 2025-02-15 10:38:12,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38439.57 MB 2025-02-15 10:38:12,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:38:12,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:38:12,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:38:12,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:12,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29580.57 MB 2025-02-15 10:38:12,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23996.98 MB 2025-02-15 10:38:12,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5583.59 MB 2025-02-15 10:38:12,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38486.93 MB 2025-02-15 10:38:12,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48064.63 MB 2025-02-15 10:38:12,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9577.69 MB 2025-02-15 10:38:12,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43977.63 MB 2025-02-15 10:38:14,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:38:14,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:38:14,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:38:14,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:14,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23996.98 MB 2025-02-15 10:38:14,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24527.82 MB 2025-02-15 10:38:14,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:38:14,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48064.63 MB 2025-02-15 10:38:14,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32891.73 MB 2025-02-15 10:38:14,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15172.89 MB 2025-02-15 10:38:14,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28506.36 MB 2025-02-15 10:38:14,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:38:14,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:38:14,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:38:14,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:14,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24527.82 MB 2025-02-15 10:38:14,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26417.35 MB 2025-02-15 10:38:14,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:38:14,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32891.73 MB 2025-02-15 10:38:14,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32891.73 MB 2025-02-15 10:38:14,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:14,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27834.78 MB 2025-02-15 10:38:15,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:38:15,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:38:15,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:38:15,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26417.35 MB 2025-02-15 10:38:15,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28659.21 MB 2025-02-15 10:38:15,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:38:15,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32891.73 MB 2025-02-15 10:38:15,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36666.61 MB 2025-02-15 10:38:15,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 10:38:15,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34203.49 MB 2025-02-15 10:38:15,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:38:15,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:38:15,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 10:38:15,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24527.82 MB 2025-02-15 10:38:15,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28659.21 MB 2025-02-15 10:38:15,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:38:15,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32891.73 MB 2025-02-15 10:38:15,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36666.61 MB 2025-02-15 10:38:15,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 10:38:15,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34203.49 MB 2025-02-15 10:38:15,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:38:15,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:38:15,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:38:15,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30192.75 MB 2025-02-15 10:38:15,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30959.75 MB 2025-02-15 10:38:15,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:38:15,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36666.61 MB 2025-02-15 10:38:15,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37079.74 MB 2025-02-15 10:38:15,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:38:15,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31667.54 MB 2025-02-15 10:38:15,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:38:15,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:38:15,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:38:15,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31372.64 MB 2025-02-15 10:38:15,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31601.85 MB 2025-02-15 10:38:15,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-15 10:38:15,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37079.74 MB 2025-02-15 10:38:15,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37079.74 MB 2025-02-15 10:38:15,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:15,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31830.61 MB 2025-02-15 10:38:15,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:38:15,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:38:15,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.12 seconds 2025-02-15 10:38:15,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18477.04 MB 2025-02-15 10:38:15,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31802.70 MB 2025-02-15 10:38:15,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13325.66 MB 2025-02-15 10:38:15,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54207.18 MB 2025-02-15 10:38:15,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37079.74 MB 2025-02-15 10:38:15,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17127.44 MB 2025-02-15 10:38:15,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31830.61 MB 2025-02-15 10:38:15,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:38:15,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:38:15,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:38:15,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31802.70 MB 2025-02-15 10:38:15,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23476.93 MB 2025-02-15 10:38:15,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8325.77 MB 2025-02-15 10:38:15,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37079.74 MB 2025-02-15 10:38:15,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37079.74 MB 2025-02-15 10:38:15,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:15,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34310.68 MB 2025-02-15 10:38:15,664 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 10:38:15,664 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:38:15,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:38:15,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:38:15,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:38:15,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:15,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23476.93 MB 2025-02-15 10:38:15,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31903.43 MB 2025-02-15 10:38:15,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 10:38:15,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37079.74 MB 2025-02-15 10:38:15,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41267.76 MB 2025-02-15 10:38:15,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 10:38:15,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31903.43 MB 2025-02-15 10:38:15,833 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 10:38:15,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:15,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:38:15,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:15,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:38:15,840 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:38:15,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:15,841 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:38:15,841 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:38:47,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:47,870 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:38:47,876 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:38:47,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:47,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 570, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:38:47,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:47,881 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 570, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:38:56,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:38:56,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:38:56,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.85 seconds 2025-02-15 10:38:56,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:56,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16940.56 MB 2025-02-15 10:38:56,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18957.76 MB 2025-02-15 10:38:56,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2017.20 MB 2025-02-15 10:38:56,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53833.89 MB 2025-02-15 10:38:56,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24503.12 MB 2025-02-15 10:38:56,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29330.77 MB 2025-02-15 10:38:56,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27770.88 MB 2025-02-15 10:38:56,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:38:56,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:38:56,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:38:56,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:56,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18957.76 MB 2025-02-15 10:38:56,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18742.15 MB 2025-02-15 10:38:56,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -215.60 MB 2025-02-15 10:38:56,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24503.12 MB 2025-02-15 10:38:56,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29569.84 MB 2025-02-15 10:38:56,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5066.72 MB 2025-02-15 10:38:56,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27116.80 MB 2025-02-15 10:38:58,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:38:58,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:38:58,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:38:58,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:58,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.15 MB 2025-02-15 10:38:58,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19272.99 MB 2025-02-15 10:38:58,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:38:58,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29569.84 MB 2025-02-15 10:38:58,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26627.54 MB 2025-02-15 10:38:58,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2942.30 MB 2025-02-15 10:38:58,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23251.54 MB 2025-02-15 10:38:58,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:38:58,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:38:58,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:38:58,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:58,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19272.99 MB 2025-02-15 10:38:58,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21162.53 MB 2025-02-15 10:38:58,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:38:58,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26627.54 MB 2025-02-15 10:38:58,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26627.54 MB 2025-02-15 10:38:58,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:58,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22579.96 MB 2025-02-15 10:38:58,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:38:58,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:38:58,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:38:58,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:58,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21162.53 MB 2025-02-15 10:38:58,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23404.38 MB 2025-02-15 10:38:58,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:38:58,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26627.54 MB 2025-02-15 10:38:58,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31346.13 MB 2025-02-15 10:38:58,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 10:38:58,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.66 MB 2025-02-15 10:38:58,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:38:58,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:38:58,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:38:58,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:58,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19272.99 MB 2025-02-15 10:38:58,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23404.38 MB 2025-02-15 10:38:58,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:38:58,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26627.54 MB 2025-02-15 10:38:58,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31346.13 MB 2025-02-15 10:38:58,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 10:38:58,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.66 MB 2025-02-15 10:38:59,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:38:59,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:38:59,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:38:59,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:59,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.93 MB 2025-02-15 10:38:59,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25704.93 MB 2025-02-15 10:38:59,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:38:59,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31346.13 MB 2025-02-15 10:38:59,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 10:38:59,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:38:59,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26412.72 MB 2025-02-15 10:38:59,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:38:59,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:38:59,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:38:59,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:59,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26117.82 MB 2025-02-15 10:38:59,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26347.71 MB 2025-02-15 10:38:59,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.90 MB 2025-02-15 10:38:59,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31759.27 MB 2025-02-15 10:38:59,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 10:38:59,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:59,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26556.57 MB 2025-02-15 10:38:59,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:38:59,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:38:59,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.24 seconds 2025-02-15 10:38:59,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:59,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.63 MB 2025-02-15 10:38:59,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26548.78 MB 2025-02-15 10:38:59,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11594.15 MB 2025-02-15 10:38:59,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53833.89 MB 2025-02-15 10:38:59,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 10:38:59,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22074.62 MB 2025-02-15 10:38:59,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26556.57 MB 2025-02-15 10:38:59,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:38:59,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:38:59,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:38:59,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:59,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26548.78 MB 2025-02-15 10:38:59,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19959.02 MB 2025-02-15 10:38:59,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6589.76 MB 2025-02-15 10:38:59,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31759.27 MB 2025-02-15 10:38:59,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-15 10:38:59,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:38:59,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29060.45 MB 2025-02-15 10:38:59,414 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:38:59,414 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:38:59,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:38:59,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:38:59,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:38:59,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:38:59,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19959.02 MB 2025-02-15 10:38:59,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28398.04 MB 2025-02-15 10:38:59,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:38:59,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31759.27 MB 2025-02-15 10:38:59,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42249.22 MB 2025-02-15 10:38:59,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:38:59,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28398.04 MB 2025-02-15 10:38:59,581 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:38:59,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:59,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:38:59,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:59,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:38:59,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:38:59,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:38:59,590 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:38:59,590 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:39:45,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:45,136 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:39:45,141 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:39:45,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:45,145 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 670, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:39:45,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:45,146 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 670, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:39:55,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:39:55,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:39:55,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.45 seconds 2025-02-15 10:39:55,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:55,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-15 10:39:55,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20008.47 MB 2025-02-15 10:39:55,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2371.09 MB 2025-02-15 10:39:55,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54834.23 MB 2025-02-15 10:39:55,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23823.65 MB 2025-02-15 10:39:55,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31010.59 MB 2025-02-15 10:39:55,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28920.68 MB 2025-02-15 10:39:55,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:39:55,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:39:55,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:39:55,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:55,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20008.47 MB 2025-02-15 10:39:55,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19260.97 MB 2025-02-15 10:39:55,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -747.49 MB 2025-02-15 10:39:55,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23823.65 MB 2025-02-15 10:39:55,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29905.39 MB 2025-02-15 10:39:55,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6081.74 MB 2025-02-15 10:39:55,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28897.84 MB 2025-02-15 10:39:57,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:39:57,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:39:57,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:39:57,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:57,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19260.97 MB 2025-02-15 10:39:57,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19791.81 MB 2025-02-15 10:39:57,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:39:57,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29905.39 MB 2025-02-15 10:39:57,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23139.98 MB 2025-02-15 10:39:57,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6765.41 MB 2025-02-15 10:39:57,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23771.40 MB 2025-02-15 10:39:57,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:39:57,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:39:57,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:39:57,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:57,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19791.81 MB 2025-02-15 10:39:57,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21681.35 MB 2025-02-15 10:39:57,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:39:57,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23139.98 MB 2025-02-15 10:39:57,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25027.41 MB 2025-02-15 10:39:57,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:39:57,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23098.78 MB 2025-02-15 10:39:57,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:39:57,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:39:57,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:39:57,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:57,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21681.35 MB 2025-02-15 10:39:57,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23923.20 MB 2025-02-15 10:39:57,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:39:57,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25027.41 MB 2025-02-15 10:39:57,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31635.54 MB 2025-02-15 10:39:57,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 10:39:57,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.53 MB 2025-02-15 10:39:57,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:39:57,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:39:57,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:39:57,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:57,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19791.81 MB 2025-02-15 10:39:57,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23923.20 MB 2025-02-15 10:39:57,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:39:57,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23139.98 MB 2025-02-15 10:39:57,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31635.54 MB 2025-02-15 10:39:57,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-15 10:39:57,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.53 MB 2025-02-15 10:39:58,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:39:58,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:39:58,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:39:58,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:58,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25457.79 MB 2025-02-15 10:39:58,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26224.80 MB 2025-02-15 10:39:58,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:39:58,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31635.54 MB 2025-02-15 10:39:58,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32048.68 MB 2025-02-15 10:39:58,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:39:58,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26932.58 MB 2025-02-15 10:39:58,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:39:58,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:39:58,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:39:58,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:58,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26637.69 MB 2025-02-15 10:39:58,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26865.88 MB 2025-02-15 10:39:58,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-15 10:39:58,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32048.68 MB 2025-02-15 10:39:58,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32048.68 MB 2025-02-15 10:39:58,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:39:58,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27087.09 MB 2025-02-15 10:39:58,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:39:58,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:39:58,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.90 seconds 2025-02-15 10:39:58,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:58,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15303.04 MB 2025-02-15 10:39:58,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27066.00 MB 2025-02-15 10:39:58,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11762.96 MB 2025-02-15 10:39:58,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54834.23 MB 2025-02-15 10:39:58,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32048.68 MB 2025-02-15 10:39:58,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22785.56 MB 2025-02-15 10:39:58,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27087.09 MB 2025-02-15 10:39:58,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:39:58,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:39:58,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:39:58,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:58,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27066.00 MB 2025-02-15 10:39:58,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20294.36 MB 2025-02-15 10:39:58,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6771.64 MB 2025-02-15 10:39:58,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32048.68 MB 2025-02-15 10:39:58,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32048.68 MB 2025-02-15 10:39:58,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:39:58,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29566.42 MB 2025-02-15 10:39:58,334 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 10:39:58,335 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:39:58,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:39:58,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:39:58,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:39:58,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:39:58,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20294.36 MB 2025-02-15 10:39:58,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28693.75 MB 2025-02-15 10:39:58,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-15 10:39:58,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32048.68 MB 2025-02-15 10:39:58,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40399.54 MB 2025-02-15 10:39:58,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 10:39:58,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28693.75 MB 2025-02-15 10:39:58,505 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 10:39:58,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:58,506 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:39:58,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:58,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:39:58,512 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:39:58,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:39:58,513 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:39:58,513 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:40:09,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:09,228 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:40:09,233 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:40:09,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:09,236 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1018, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:40:09,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:09,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1018, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:40:25,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:40:25,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:40:25,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.92 seconds 2025-02-15 10:40:25,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:25,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20062.29 MB 2025-02-15 10:40:25,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23665.20 MB 2025-02-15 10:40:25,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3602.91 MB 2025-02-15 10:40:25,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48750.40 MB 2025-02-15 10:40:25,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28085.06 MB 2025-02-15 10:40:25,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20665.34 MB 2025-02-15 10:40:25,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.07 MB 2025-02-15 10:40:25,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:40:25,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:40:25,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:40:25,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:25,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23665.20 MB 2025-02-15 10:40:25,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21071.16 MB 2025-02-15 10:40:25,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2594.04 MB 2025-02-15 10:40:25,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28085.06 MB 2025-02-15 10:40:25,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37081.84 MB 2025-02-15 10:40:25,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8996.78 MB 2025-02-15 10:40:25,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34760.39 MB 2025-02-15 10:40:27,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:40:27,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:40:27,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:40:27,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21071.16 MB 2025-02-15 10:40:27,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21602.00 MB 2025-02-15 10:40:27,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:40:27,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37081.84 MB 2025-02-15 10:40:27,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-15 10:40:27,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10475.27 MB 2025-02-15 10:40:27,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25580.55 MB 2025-02-15 10:40:27,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:40:27,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:40:27,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:40:27,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.00 MB 2025-02-15 10:40:27,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23491.54 MB 2025-02-15 10:40:27,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:40:27,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-15 10:40:27,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27550.29 MB 2025-02-15 10:40:27,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:40:27,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24908.97 MB 2025-02-15 10:40:27,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:40:27,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:40:27,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:40:27,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23491.54 MB 2025-02-15 10:40:27,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25733.39 MB 2025-02-15 10:40:27,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:40:27,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27550.29 MB 2025-02-15 10:40:27,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33212.60 MB 2025-02-15 10:40:27,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:40:27,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31277.68 MB 2025-02-15 10:40:27,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:40:27,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:40:27,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:40:27,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.00 MB 2025-02-15 10:40:27,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25733.39 MB 2025-02-15 10:40:27,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:40:27,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-15 10:40:27,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33212.60 MB 2025-02-15 10:40:27,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:40:27,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31277.68 MB 2025-02-15 10:40:27,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:40:27,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:40:27,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:40:27,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27266.94 MB 2025-02-15 10:40:27,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28033.94 MB 2025-02-15 10:40:27,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:40:27,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33212.60 MB 2025-02-15 10:40:27,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-15 10:40:27,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:40:27,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28741.73 MB 2025-02-15 10:40:27,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:40:27,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:40:27,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:40:27,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28446.83 MB 2025-02-15 10:40:27,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28676.83 MB 2025-02-15 10:40:27,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.01 MB 2025-02-15 10:40:27,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-15 10:40:27,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-15 10:40:27,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:40:27,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28885.82 MB 2025-02-15 10:40:27,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:40:27,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:40:27,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.37 seconds 2025-02-15 10:40:27,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.50 MB 2025-02-15 10:40:27,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28877.91 MB 2025-02-15 10:40:27,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12362.41 MB 2025-02-15 10:40:27,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48750.40 MB 2025-02-15 10:40:27,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-15 10:40:27,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15122.56 MB 2025-02-15 10:40:27,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28885.82 MB 2025-02-15 10:40:27,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:40:27,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:40:27,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:40:27,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28877.91 MB 2025-02-15 10:40:27,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21519.89 MB 2025-02-15 10:40:27,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7358.02 MB 2025-02-15 10:40:27,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-15 10:40:27,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-15 10:40:27,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:40:27,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31389.57 MB 2025-02-15 10:40:27,899 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:40:27,899 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:40:27,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:40:27,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:40:27,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:40:27,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:27,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21519.89 MB 2025-02-15 10:40:27,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.91 MB 2025-02-15 10:40:27,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:40:27,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-15 10:40:27,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42018.54 MB 2025-02-15 10:40:27,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:40:27,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29958.91 MB 2025-02-15 10:40:28,069 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:40:28,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:28,070 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:40:28,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:28,071 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:40:28,076 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:40:28,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:28,079 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:40:28,079 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:40:40,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:40,035 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:40:40,043 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:40:40,050 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:40,050 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 178, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:40:40,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:40,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 178, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:40:42,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:40:42,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:40:42,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-15 10:40:42,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:42,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14209.04 MB 2025-02-15 10:40:42,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14838.97 MB 2025-02-15 10:40:42,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 629.93 MB 2025-02-15 10:40:42,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54603.55 MB 2025-02-15 10:40:42,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-15 10:40:42,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34833.69 MB 2025-02-15 10:40:42,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23680.41 MB 2025-02-15 10:40:42,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:40:42,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:40:42,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:40:42,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:42,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14838.97 MB 2025-02-15 10:40:42,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15137.08 MB 2025-02-15 10:40:42,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.11 MB 2025-02-15 10:40:42,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-15 10:40:42,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-15 10:40:42,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:40:42,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17325.12 MB 2025-02-15 10:40:43,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:40:43,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:40:43,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.89 seconds 2025-02-15 10:40:43,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:43,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15137.08 MB 2025-02-15 10:40:43,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15371.98 MB 2025-02-15 10:40:43,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 10:40:43,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-15 10:40:43,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18440.26 MB 2025-02-15 10:40:43,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1329.59 MB 2025-02-15 10:40:43,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19307.77 MB 2025-02-15 10:40:43,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:40:43,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:40:43,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:40:43,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:43,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15371.98 MB 2025-02-15 10:40:43,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16207.90 MB 2025-02-15 10:40:43,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 10:40:43,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18440.26 MB 2025-02-15 10:40:43,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18859.69 MB 2025-02-15 10:40:43,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 10:40:43,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16835.11 MB 2025-02-15 10:40:44,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:40:44,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:40:44,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:40:44,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16207.90 MB 2025-02-15 10:40:44,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17199.95 MB 2025-02-15 10:40:44,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 10:40:44,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18859.69 MB 2025-02-15 10:40:44,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21376.27 MB 2025-02-15 10:40:44,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-15 10:40:44,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19653.56 MB 2025-02-15 10:40:44,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:40:44,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:40:44,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:40:44,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15371.98 MB 2025-02-15 10:40:44,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17199.95 MB 2025-02-15 10:40:44,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 10:40:44,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18440.26 MB 2025-02-15 10:40:44,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21376.27 MB 2025-02-15 10:40:44,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-15 10:40:44,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19653.56 MB 2025-02-15 10:40:44,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:40:44,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:40:44,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 10:40:44,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17878.84 MB 2025-02-15 10:40:44,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18218.24 MB 2025-02-15 10:40:44,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-15 10:40:44,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21376.27 MB 2025-02-15 10:40:44,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21556.63 MB 2025-02-15 10:40:44,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 10:40:44,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18538.57 MB 2025-02-15 10:40:44,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:40:44,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:40:44,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:40:44,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18400.95 MB 2025-02-15 10:40:44,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18627.69 MB 2025-02-15 10:40:44,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.74 MB 2025-02-15 10:40:44,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21556.63 MB 2025-02-15 10:40:44,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21556.63 MB 2025-02-15 10:40:44,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:40:44,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18642.31 MB 2025-02-15 10:40:44,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:40:44,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:40:44,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.10 seconds 2025-02-15 10:40:44,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13588.87 MB 2025-02-15 10:40:44,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18828.54 MB 2025-02-15 10:40:44,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5239.67 MB 2025-02-15 10:40:44,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54603.55 MB 2025-02-15 10:40:44,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21556.63 MB 2025-02-15 10:40:44,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33046.92 MB 2025-02-15 10:40:44,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18828.54 MB 2025-02-15 10:40:44,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:40:44,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:40:44,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:40:44,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18828.54 MB 2025-02-15 10:40:44,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17535.33 MB 2025-02-15 10:40:44,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1293.22 MB 2025-02-15 10:40:44,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21556.63 MB 2025-02-15 10:40:44,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21556.63 MB 2025-02-15 10:40:44,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:40:44,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19062.85 MB 2025-02-15 10:40:44,470 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 10:40:44,470 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 10:40:44,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:40:44,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:40:44,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:40:44,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:40:44,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17535.33 MB 2025-02-15 10:40:44,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25958.53 MB 2025-02-15 10:40:44,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 10:40:44,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21556.63 MB 2025-02-15 10:40:44,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32027.71 MB 2025-02-15 10:40:44,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 10:40:44,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25958.53 MB 2025-02-15 10:40:44,736 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 10:40:44,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:44,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:40:44,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:44,741 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:40:44,749 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:40:44,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:40:44,751 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:40:44,751 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 10:41:18,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:18,071 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:41:18,076 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:41:18,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:18,080 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:41:18,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:18,081 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:41:21,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:41:21,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:41:21,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.19 seconds 2025-02-15 10:41:21,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:21,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-15 10:41:21,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-15 10:41:21,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-15 10:41:21,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40403.73 MB 2025-02-15 10:41:21,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19795.02 MB 2025-02-15 10:41:21,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20608.71 MB 2025-02-15 10:41:21,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.07 MB 2025-02-15 10:41:21,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:41:21,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:41:21,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:41:21,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:21,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-15 10:41:21,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15384.68 MB 2025-02-15 10:41:21,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.53 MB 2025-02-15 10:41:21,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19795.02 MB 2025-02-15 10:41:21,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19795.02 MB 2025-02-15 10:41:21,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:41:21,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17823.11 MB 2025-02-15 10:41:22,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:41:22,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:41:22,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-15 10:41:22,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15384.68 MB 2025-02-15 10:41:22,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15640.81 MB 2025-02-15 10:41:22,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 10:41:22,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19795.02 MB 2025-02-15 10:41:22,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-15 10:41:22,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -419.43 MB 2025-02-15 10:41:22,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19641.34 MB 2025-02-15 10:41:22,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:41:22,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:41:22,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:41:22,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 10:41:22,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.23 MB 2025-02-15 10:41:22,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 10:41:22,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-15 10:41:22,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-15 10:41:22,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:41:22,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.14 MB 2025-02-15 10:41:22,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:41:22,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:41:22,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:41:22,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.23 MB 2025-02-15 10:41:22,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 10:41:22,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 10:41:22,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-15 10:41:22,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21661.48 MB 2025-02-15 10:41:22,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 10:41:22,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 10:41:22,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:41:22,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:41:22,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 10:41:22,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-15 10:41:22,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-15 10:41:22,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 10:41:22,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-15 10:41:22,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21661.48 MB 2025-02-15 10:41:22,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 10:41:22,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.04 MB 2025-02-15 10:41:22,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:41:22,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:41:22,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:41:22,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.89 MB 2025-02-15 10:41:22,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18743.97 MB 2025-02-15 10:41:22,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 10:41:22,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21661.48 MB 2025-02-15 10:41:22,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21858.62 MB 2025-02-15 10:41:22,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-15 10:41:22,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19089.76 MB 2025-02-15 10:41:22,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:41:22,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:41:22,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:41:22,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18943.20 MB 2025-02-15 10:41:22,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19169.72 MB 2025-02-15 10:41:22,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.52 MB 2025-02-15 10:41:22,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21858.62 MB 2025-02-15 10:41:22,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21860.71 MB 2025-02-15 10:41:22,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 10:41:22,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19205.58 MB 2025-02-15 10:41:22,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:41:22,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:41:22,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.50 seconds 2025-02-15 10:41:22,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-15 10:41:22,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19370.79 MB 2025-02-15 10:41:22,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5691.33 MB 2025-02-15 10:41:22,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40403.73 MB 2025-02-15 10:41:22,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21860.71 MB 2025-02-15 10:41:22,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18543.02 MB 2025-02-15 10:41:22,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19370.79 MB 2025-02-15 10:41:22,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:41:22,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:41:22,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:41:22,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19370.79 MB 2025-02-15 10:41:22,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17706.40 MB 2025-02-15 10:41:22,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1664.39 MB 2025-02-15 10:41:22,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21860.71 MB 2025-02-15 10:41:22,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21860.71 MB 2025-02-15 10:41:22,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:41:22,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19370.79 MB 2025-02-15 10:41:22,899 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:41:22,899 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:41:22,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:41:22,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:41:22,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:41:22,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:41:22,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17706.40 MB 2025-02-15 10:41:22,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26145.42 MB 2025-02-15 10:41:22,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:41:22,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21860.71 MB 2025-02-15 10:41:22,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32350.67 MB 2025-02-15 10:41:22,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:41:22,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26145.42 MB 2025-02-15 10:41:23,165 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:41:23,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:23,168 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:41:23,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:23,170 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:41:23,178 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:41:23,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:41:23,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:41:23,180 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:42:29,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:29,228 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:42:29,233 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:42:29,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:29,238 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 658, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:42:29,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:29,239 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 658, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:42:39,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:42:39,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:42:39,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.14 seconds 2025-02-15 10:42:39,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:39,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17553.76 MB 2025-02-15 10:42:39,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19882.38 MB 2025-02-15 10:42:39,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2328.63 MB 2025-02-15 10:42:39,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44935.68 MB 2025-02-15 10:42:39,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24410.85 MB 2025-02-15 10:42:39,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20524.83 MB 2025-02-15 10:42:39,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28837.07 MB 2025-02-15 10:42:39,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:42:39,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:42:39,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 10:42:39,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:39,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19882.38 MB 2025-02-15 10:42:39,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19198.59 MB 2025-02-15 10:42:39,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -683.79 MB 2025-02-15 10:42:39,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24410.85 MB 2025-02-15 10:42:39,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30226.25 MB 2025-02-15 10:42:39,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5815.40 MB 2025-02-15 10:42:39,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28503.98 MB 2025-02-15 10:42:41,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:42:41,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:42:41,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:42:41,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19198.59 MB 2025-02-15 10:42:41,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19729.43 MB 2025-02-15 10:42:41,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:42:41,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30226.25 MB 2025-02-15 10:42:41,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25826.43 MB 2025-02-15 10:42:41,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4399.82 MB 2025-02-15 10:42:41,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23707.98 MB 2025-02-15 10:42:41,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:42:41,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:42:41,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:42:41,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-15 10:42:41,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21618.96 MB 2025-02-15 10:42:41,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:42:41,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25826.43 MB 2025-02-15 10:42:41,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25826.43 MB 2025-02-15 10:42:41,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:42:41,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23036.39 MB 2025-02-15 10:42:41,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:42:41,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:42:41,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:42:41,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21618.96 MB 2025-02-15 10:42:41,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-15 10:42:41,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:42:41,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25826.43 MB 2025-02-15 10:42:41,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31488.74 MB 2025-02-15 10:42:41,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:42:41,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-15 10:42:41,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:42:41,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:42:41,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 10:42:41,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-15 10:42:41,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-15 10:42:41,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:42:41,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25826.43 MB 2025-02-15 10:42:41,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31488.74 MB 2025-02-15 10:42:41,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:42:41,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-15 10:42:41,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:42:41,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:42:41,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:42:41,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25394.36 MB 2025-02-15 10:42:41,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26161.36 MB 2025-02-15 10:42:41,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:42:41,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31488.74 MB 2025-02-15 10:42:41,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31903.97 MB 2025-02-15 10:42:41,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:42:41,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.15 MB 2025-02-15 10:42:41,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:42:41,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:42:41,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:42:41,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26574.25 MB 2025-02-15 10:42:41,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26802.08 MB 2025-02-15 10:42:41,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.83 MB 2025-02-15 10:42:41,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31903.97 MB 2025-02-15 10:42:41,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31903.97 MB 2025-02-15 10:42:41,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:42:41,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26992.07 MB 2025-02-15 10:42:41,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:42:41,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:42:41,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.56 seconds 2025-02-15 10:42:41,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:41,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15261.23 MB 2025-02-15 10:42:41,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.22 MB 2025-02-15 10:42:41,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11740.99 MB 2025-02-15 10:42:41,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44935.68 MB 2025-02-15 10:42:41,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31903.97 MB 2025-02-15 10:42:41,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13031.70 MB 2025-02-15 10:42:41,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27002.22 MB 2025-02-15 10:42:42,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:42:42,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:42:42,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:42:42,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:42,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27002.22 MB 2025-02-15 10:42:42,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20251.46 MB 2025-02-15 10:42:42,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6750.76 MB 2025-02-15 10:42:42,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31903.97 MB 2025-02-15 10:42:42,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31903.97 MB 2025-02-15 10:42:42,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:42:42,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29502.22 MB 2025-02-15 10:42:42,089 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 10:42:42,089 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:42:42,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:42:42,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:42:42,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:42:42,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:42:42,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20251.46 MB 2025-02-15 10:42:42,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28652.33 MB 2025-02-15 10:42:42,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 10:42:42,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31903.97 MB 2025-02-15 10:42:42,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40254.83 MB 2025-02-15 10:42:42,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 10:42:42,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28652.33 MB 2025-02-15 10:42:42,257 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 10:42:42,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:42,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:42:42,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:42,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:42:42,265 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:42:42,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:42:42,266 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:42:42,266 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:43:43,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:43:43,643 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:43:43,648 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:43:43,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:43:43,652 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1631, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:43:43,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:43:43,653 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1631, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:44:08,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:44:08,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:44:08,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.11 seconds 2025-02-15 10:44:08,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:08,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24333.78 MB 2025-02-15 10:44:08,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30105.79 MB 2025-02-15 10:44:08,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5772.02 MB 2025-02-15 10:44:08,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48605.69 MB 2025-02-15 10:44:08,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38975.57 MB 2025-02-15 10:44:08,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9630.12 MB 2025-02-15 10:44:08,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39014.47 MB 2025-02-15 10:44:08,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:44:08,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:44:08,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:44:08,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:08,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30105.79 MB 2025-02-15 10:44:08,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.91 MB 2025-02-15 10:44:08,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5848.88 MB 2025-02-15 10:44:08,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38975.57 MB 2025-02-15 10:44:08,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51449.43 MB 2025-02-15 10:44:08,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12473.86 MB 2025-02-15 10:44:08,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46731.37 MB 2025-02-15 10:44:10,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:44:10,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:44:10,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 10:44:10,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:10,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24256.91 MB 2025-02-15 10:44:10,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.75 MB 2025-02-15 10:44:10,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:44:10,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51449.43 MB 2025-02-15 10:44:10,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30442.26 MB 2025-02-15 10:44:10,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21007.17 MB 2025-02-15 10:44:10,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28766.30 MB 2025-02-15 10:44:10,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:44:10,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:44:10,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:44:10,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:10,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-15 10:44:10,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26677.29 MB 2025-02-15 10:44:10,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:44:10,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 10:44:10,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30442.26 MB 2025-02-15 10:44:10,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:10,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28094.72 MB 2025-02-15 10:44:11,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:44:11,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:44:11,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:44:11,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26677.29 MB 2025-02-15 10:44:11,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-15 10:44:11,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:44:11,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 10:44:11,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36576.43 MB 2025-02-15 10:44:11,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:44:11,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-15 10:44:11,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:44:11,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:44:11,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:44:11,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-15 10:44:11,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-15 10:44:11,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:44:11,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30442.26 MB 2025-02-15 10:44:11,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36576.43 MB 2025-02-15 10:44:11,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:44:11,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-15 10:44:11,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:44:11,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:44:11,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:44:11,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30452.69 MB 2025-02-15 10:44:11,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.69 MB 2025-02-15 10:44:11,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:44:11,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36576.43 MB 2025-02-15 10:44:11,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36989.57 MB 2025-02-15 10:44:11,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:44:11,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31927.48 MB 2025-02-15 10:44:11,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:44:11,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:44:11,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:44:11,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31632.58 MB 2025-02-15 10:44:11,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31861.36 MB 2025-02-15 10:44:11,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 10:44:11,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36989.57 MB 2025-02-15 10:44:11,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36989.57 MB 2025-02-15 10:44:11,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:11,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32070.60 MB 2025-02-15 10:44:11,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:44:11,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:44:11,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.64 seconds 2025-02-15 10:44:11,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18651.24 MB 2025-02-15 10:44:11,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32062.22 MB 2025-02-15 10:44:11,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13410.98 MB 2025-02-15 10:44:11,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48605.69 MB 2025-02-15 10:44:11,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36989.57 MB 2025-02-15 10:44:11,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11616.12 MB 2025-02-15 10:44:11,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32070.60 MB 2025-02-15 10:44:11,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:44:11,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:44:11,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:44:11,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32062.22 MB 2025-02-15 10:44:11,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23650.06 MB 2025-02-15 10:44:11,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8412.15 MB 2025-02-15 10:44:11,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36989.57 MB 2025-02-15 10:44:11,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36989.57 MB 2025-02-15 10:44:11,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:11,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34569.28 MB 2025-02-15 10:44:11,579 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 10:44:11,579 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:44:11,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:44:11,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:44:11,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:44:11,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:11,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23650.06 MB 2025-02-15 10:44:11,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32073.27 MB 2025-02-15 10:44:11,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 10:44:11,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36989.57 MB 2025-02-15 10:44:11,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45365.59 MB 2025-02-15 10:44:11,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 10:44:11,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32073.27 MB 2025-02-15 10:44:11,750 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 10:44:11,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:11,752 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:44:11,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:11,753 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:44:11,758 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:44:11,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:11,759 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:44:11,759 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:44:18,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:18,450 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:44:18,457 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:44:18,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:18,463 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1624, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:44:18,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:18,465 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1624, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:44:43,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:44:43,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:44:43,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.38 seconds 2025-02-15 10:44:43,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:43,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24285.00 MB 2025-02-15 10:44:43,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30033.29 MB 2025-02-15 10:44:43,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5748.29 MB 2025-02-15 10:44:43,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53741.62 MB 2025-02-15 10:44:43,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38954.60 MB 2025-02-15 10:44:43,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14787.02 MB 2025-02-15 10:44:43,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38965.69 MB 2025-02-15 10:44:43,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:44:43,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:44:43,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:44:43,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:43,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30033.29 MB 2025-02-15 10:44:43,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24220.52 MB 2025-02-15 10:44:43,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5812.77 MB 2025-02-15 10:44:43,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38954.60 MB 2025-02-15 10:44:43,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45642.42 MB 2025-02-15 10:44:43,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6687.82 MB 2025-02-15 10:44:43,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41303.09 MB 2025-02-15 10:44:45,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:44:45,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:44:45,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:44:45,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:45,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24220.52 MB 2025-02-15 10:44:45,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24751.36 MB 2025-02-15 10:44:45,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:44:45,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45642.42 MB 2025-02-15 10:44:45,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30446.45 MB 2025-02-15 10:44:45,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15195.96 MB 2025-02-15 10:44:45,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28729.91 MB 2025-02-15 10:44:45,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:44:45,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:44:45,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:44:45,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:45,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-15 10:44:45,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26640.90 MB 2025-02-15 10:44:45,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:44:45,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30446.45 MB 2025-02-15 10:44:45,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30446.45 MB 2025-02-15 10:44:45,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:45,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28058.32 MB 2025-02-15 10:44:46,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:44:46,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:44:46,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:44:46,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26640.90 MB 2025-02-15 10:44:46,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-15 10:44:46,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:44:46,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30446.45 MB 2025-02-15 10:44:46,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36580.62 MB 2025-02-15 10:44:46,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:44:46,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-15 10:44:46,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:44:46,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:44:46,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:44:46,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-15 10:44:46,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-15 10:44:46,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:44:46,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30446.45 MB 2025-02-15 10:44:46,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36580.62 MB 2025-02-15 10:44:46,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:44:46,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-15 10:44:46,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:44:46,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:44:46,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:44:46,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30416.29 MB 2025-02-15 10:44:46,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.30 MB 2025-02-15 10:44:46,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:44:46,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36580.62 MB 2025-02-15 10:44:46,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36995.86 MB 2025-02-15 10:44:46,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:44:46,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.08 MB 2025-02-15 10:44:46,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:44:46,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:44:46,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:44:46,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31596.19 MB 2025-02-15 10:44:46,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31823.49 MB 2025-02-15 10:44:46,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.30 MB 2025-02-15 10:44:46,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36995.86 MB 2025-02-15 10:44:46,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36995.86 MB 2025-02-15 10:44:46,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:46,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32037.95 MB 2025-02-15 10:44:46,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:44:46,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:44:46,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.87 seconds 2025-02-15 10:44:46,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18626.85 MB 2025-02-15 10:44:46,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32024.56 MB 2025-02-15 10:44:46,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13397.71 MB 2025-02-15 10:44:46,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53741.62 MB 2025-02-15 10:44:46,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36995.86 MB 2025-02-15 10:44:46,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16745.76 MB 2025-02-15 10:44:46,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32037.95 MB 2025-02-15 10:44:46,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:44:46,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:44:46,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:44:46,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32024.56 MB 2025-02-15 10:44:46,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23631.24 MB 2025-02-15 10:44:46,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8393.32 MB 2025-02-15 10:44:46,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36995.86 MB 2025-02-15 10:44:46,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36995.86 MB 2025-02-15 10:44:46,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:44:46,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34536.23 MB 2025-02-15 10:44:46,625 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:44:46,625 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:44:46,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:44:46,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:44:46,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:44:46,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:44:46,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23631.24 MB 2025-02-15 10:44:46,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32070.26 MB 2025-02-15 10:44:46,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:44:46,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36995.86 MB 2025-02-15 10:44:46,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45386.56 MB 2025-02-15 10:44:46,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:44:46,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32070.26 MB 2025-02-15 10:44:46,792 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:44:46,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:46,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:44:46,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:46,795 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:44:46,800 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:44:46,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:44:46,801 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:44:46,801 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:45:05,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:05,194 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:45:05,202 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:45:05,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:05,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 78, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:45:05,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:05,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 78, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:45:06,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:45:06,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:45:06,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.30 seconds 2025-02-15 10:45:06,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:06,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13512.22 MB 2025-02-15 10:45:06,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13788.26 MB 2025-02-15 10:45:06,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-15 10:45:06,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57971.57 MB 2025-02-15 10:45:06,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:06,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41018.20 MB 2025-02-15 10:45:06,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22757.10 MB 2025-02-15 10:45:06,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:45:06,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:45:06,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:45:06,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:06,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13788.26 MB 2025-02-15 10:45:06,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13922.00 MB 2025-02-15 10:45:06,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 133.74 MB 2025-02-15 10:45:06,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:06,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:06,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:06,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14336.12 MB 2025-02-15 10:45:06,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:45:06,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:45:06,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.41 seconds 2025-02-15 10:45:06,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:06,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13922.00 MB 2025-02-15 10:45:06,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14025.51 MB 2025-02-15 10:45:06,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 103.51 MB 2025-02-15 10:45:06,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:06,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:06,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:06,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18008.79 MB 2025-02-15 10:45:06,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:45:06,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:45:06,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:45:06,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:06,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.45 MB 2025-02-15 10:45:06,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14393.82 MB 2025-02-15 10:45:06,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 368.37 MB 2025-02-15 10:45:06,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:06,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:06,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:06,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14670.22 MB 2025-02-15 10:45:07,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:45:07,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:45:07,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:45:07,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14393.82 MB 2025-02-15 10:45:07,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14841.26 MB 2025-02-15 10:45:07,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 447.44 MB 2025-02-15 10:45:07,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:07,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:07,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:07,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15912.12 MB 2025-02-15 10:45:07,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:45:07,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:45:07,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 10:45:07,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.45 MB 2025-02-15 10:45:07,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14841.26 MB 2025-02-15 10:45:07,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 815.81 MB 2025-02-15 10:45:07,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:07,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16953.38 MB 2025-02-15 10:45:07,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:07,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15912.12 MB 2025-02-15 10:45:07,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:45:07,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:45:07,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:45:07,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15273.20 MB 2025-02-15 10:45:07,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15461.11 MB 2025-02-15 10:45:07,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.90 MB 2025-02-15 10:45:07,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16953.38 MB 2025-02-15 10:45:07,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17068.72 MB 2025-02-15 10:45:07,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 115.34 MB 2025-02-15 10:45:07,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15599.13 MB 2025-02-15 10:45:07,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:45:07,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:45:07,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:45:07,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15579.97 MB 2025-02-15 10:45:07,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15767.27 MB 2025-02-15 10:45:07,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.30 MB 2025-02-15 10:45:07,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17068.72 MB 2025-02-15 10:45:07,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17068.72 MB 2025-02-15 10:45:07,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:07,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15767.27 MB 2025-02-15 10:45:07,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:45:07,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:45:07,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 10:45:07,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13240.46 MB 2025-02-15 10:45:07,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15934.23 MB 2025-02-15 10:45:07,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2693.77 MB 2025-02-15 10:45:07,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57971.57 MB 2025-02-15 10:45:07,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17068.72 MB 2025-02-15 10:45:07,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40902.85 MB 2025-02-15 10:45:07,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15934.23 MB 2025-02-15 10:45:07,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:45:07,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:45:07,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 10:45:07,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15934.23 MB 2025-02-15 10:45:07,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16196.80 MB 2025-02-15 10:45:07,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.56 MB 2025-02-15 10:45:07,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17068.72 MB 2025-02-15 10:45:07,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17408.46 MB 2025-02-15 10:45:07,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 339.74 MB 2025-02-15 10:45:07,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16447.04 MB 2025-02-15 10:45:07,395 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6775, cut from 6777 2025-02-15 10:45:07,395 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:45:07,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:45:07,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:45:07,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:45:07,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:07,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16196.80 MB 2025-02-15 10:45:07,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23205.25 MB 2025-02-15 10:45:07,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7008.45 MB 2025-02-15 10:45:07,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17408.46 MB 2025-02-15 10:45:07,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26117.93 MB 2025-02-15 10:45:07,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8709.47 MB 2025-02-15 10:45:07,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23205.25 MB 2025-02-15 10:45:07,620 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6567] 2025-02-15 10:45:07,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:07,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:45:07,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:07,624 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:45:07,632 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:45:07,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:07,634 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:45:07,634 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 10:45:18,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:18,145 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:45:18,150 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:45:18,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:18,153 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 349, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:45:18,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:18,154 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 349, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:45:23,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:45:23,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:45:23,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.42 seconds 2025-02-15 10:45:23,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:23,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15400.59 MB 2025-02-15 10:45:23,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16635.82 MB 2025-02-15 10:45:23,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1235.22 MB 2025-02-15 10:45:23,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33084.67 MB 2025-02-15 10:45:23,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19457.38 MB 2025-02-15 10:45:23,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13627.29 MB 2025-02-15 10:45:23,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25552.25 MB 2025-02-15 10:45:23,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:45:23,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:45:23,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:45:23,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:23,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16635.82 MB 2025-02-15 10:45:23,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17142.72 MB 2025-02-15 10:45:23,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 506.90 MB 2025-02-15 10:45:23,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19457.38 MB 2025-02-15 10:45:23,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23007.85 MB 2025-02-15 10:45:23,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3550.48 MB 2025-02-15 10:45:23,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21356.94 MB 2025-02-15 10:45:25,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:45:25,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:45:25,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.64 seconds 2025-02-15 10:45:25,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17142.72 MB 2025-02-15 10:45:25,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17588.63 MB 2025-02-15 10:45:25,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 445.91 MB 2025-02-15 10:45:25,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23007.85 MB 2025-02-15 10:45:25,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19411.24 MB 2025-02-15 10:45:25,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3596.62 MB 2025-02-15 10:45:25,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21568.21 MB 2025-02-15 10:45:25,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:45:25,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:45:25,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:45:25,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17588.63 MB 2025-02-15 10:45:25,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19176.17 MB 2025-02-15 10:45:25,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1587.54 MB 2025-02-15 10:45:25,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19411.24 MB 2025-02-15 10:45:25,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22185.77 MB 2025-02-15 10:45:25,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2774.53 MB 2025-02-15 10:45:25,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20366.81 MB 2025-02-15 10:45:25,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:45:25,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:45:25,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:45:25,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19176.17 MB 2025-02-15 10:45:25,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21059.34 MB 2025-02-15 10:45:25,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1883.17 MB 2025-02-15 10:45:25,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22185.77 MB 2025-02-15 10:45:25,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27734.84 MB 2025-02-15 10:45:25,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5549.06 MB 2025-02-15 10:45:25,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25717.51 MB 2025-02-15 10:45:25,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:45:25,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:45:25,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 10:45:25,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17588.63 MB 2025-02-15 10:45:25,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21059.34 MB 2025-02-15 10:45:25,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3470.71 MB 2025-02-15 10:45:25,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19411.24 MB 2025-02-15 10:45:25,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27734.84 MB 2025-02-15 10:45:25,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8323.60 MB 2025-02-15 10:45:25,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25717.51 MB 2025-02-15 10:45:25,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:45:25,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:45:25,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:45:25,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.50 MB 2025-02-15 10:45:25,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22992.78 MB 2025-02-15 10:45:25,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.28 MB 2025-02-15 10:45:25,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27734.84 MB 2025-02-15 10:45:25,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-15 10:45:25,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 348.13 MB 2025-02-15 10:45:25,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23587.32 MB 2025-02-15 10:45:25,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:45:25,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:45:25,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:45:25,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23339.61 MB 2025-02-15 10:45:25,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23567.11 MB 2025-02-15 10:45:25,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.50 MB 2025-02-15 10:45:25,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28082.96 MB 2025-02-15 10:45:25,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-15 10:45:25,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:25,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23720.12 MB 2025-02-15 10:45:25,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:45:25,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:45:25,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.44 seconds 2025-02-15 10:45:25,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14184.65 MB 2025-02-15 10:45:25,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23768.18 MB 2025-02-15 10:45:25,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9583.53 MB 2025-02-15 10:45:25,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33084.67 MB 2025-02-15 10:45:25,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-15 10:45:25,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5001.71 MB 2025-02-15 10:45:25,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23768.18 MB 2025-02-15 10:45:25,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:45:25,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:45:25,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:45:25,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23768.18 MB 2025-02-15 10:45:25,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26782.22 MB 2025-02-15 10:45:25,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:45:25,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28082.96 MB 2025-02-15 10:45:25,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-15 10:45:25,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:45:25,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27083.59 MB 2025-02-15 10:45:25,888 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:45:25,888 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:45:25,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:45:25,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:45:25,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:45:25,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:45:25,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18887.99 MB 2025-02-15 10:45:25,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27327.01 MB 2025-02-15 10:45:25,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:45:25,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28082.96 MB 2025-02-15 10:45:25,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38572.92 MB 2025-02-15 10:45:25,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:45:25,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27327.01 MB 2025-02-15 10:45:26,052 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:45:26,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:26,054 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:45:26,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:26,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:45:26,059 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:45:26,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:45:26,060 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:45:26,060 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:46:46,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:46,563 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:46:46,568 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:46:46,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:46,572 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:46:46,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:46,573 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:46:49,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:46:49,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:46:49,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-15 10:46:49,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:49,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14425.05 MB 2025-02-15 10:46:49,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15164.69 MB 2025-02-15 10:46:49,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.64 MB 2025-02-15 10:46:49,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51157.93 MB 2025-02-15 10:46:49,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18461.23 MB 2025-02-15 10:46:49,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32696.70 MB 2025-02-15 10:46:49,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24122.92 MB 2025-02-15 10:46:49,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:46:49,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:46:49,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:46:49,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:49,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15164.69 MB 2025-02-15 10:46:49,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15523.63 MB 2025-02-15 10:46:49,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 358.94 MB 2025-02-15 10:46:49,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18461.23 MB 2025-02-15 10:46:49,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19941.82 MB 2025-02-15 10:46:49,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1480.59 MB 2025-02-15 10:46:49,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18101.62 MB 2025-02-15 10:46:50,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:46:50,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:46:50,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-15 10:46:50,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:50,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15523.63 MB 2025-02-15 10:46:50,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15801.00 MB 2025-02-15 10:46:50,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-15 10:46:50,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19941.82 MB 2025-02-15 10:46:50,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18805.16 MB 2025-02-15 10:46:50,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1136.66 MB 2025-02-15 10:46:50,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.26 MB 2025-02-15 10:46:50,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:46:50,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:46:50,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:46:50,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:50,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15801.00 MB 2025-02-15 10:46:50,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.04 MB 2025-02-15 10:46:50,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-15 10:46:50,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18805.16 MB 2025-02-15 10:46:50,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19300.09 MB 2025-02-15 10:46:50,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 494.93 MB 2025-02-15 10:46:50,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17528.65 MB 2025-02-15 10:46:50,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:46:50,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:46:50,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:46:50,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:50,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16788.04 MB 2025-02-15 10:46:50,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17959.44 MB 2025-02-15 10:46:50,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.40 MB 2025-02-15 10:46:50,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19300.09 MB 2025-02-15 10:46:50,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22269.66 MB 2025-02-15 10:46:50,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-15 10:46:50,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20856.30 MB 2025-02-15 10:46:50,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:46:50,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:46:50,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 10:46:50,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:50,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15801.00 MB 2025-02-15 10:46:50,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17959.44 MB 2025-02-15 10:46:50,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.44 MB 2025-02-15 10:46:50,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18805.16 MB 2025-02-15 10:46:50,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22269.66 MB 2025-02-15 10:46:50,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3464.50 MB 2025-02-15 10:46:50,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20856.30 MB 2025-02-15 10:46:51,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:46:51,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:46:51,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:46:51,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:51,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18760.72 MB 2025-02-15 10:46:51,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19161.48 MB 2025-02-15 10:46:51,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.76 MB 2025-02-15 10:46:51,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22269.66 MB 2025-02-15 10:46:51,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22483.57 MB 2025-02-15 10:46:51,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 10:46:51,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19534.97 MB 2025-02-15 10:46:51,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:46:51,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:46:51,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:46:51,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:51,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19377.22 MB 2025-02-15 10:46:51,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19606.00 MB 2025-02-15 10:46:51,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 10:46:51,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22483.57 MB 2025-02-15 10:46:51,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22483.57 MB 2025-02-15 10:46:51,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:46:51,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19671.14 MB 2025-02-15 10:46:51,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:46:51,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:46:51,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.50 seconds 2025-02-15 10:46:51,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:51,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13696.88 MB 2025-02-15 10:46:51,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19807.08 MB 2025-02-15 10:46:51,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6110.20 MB 2025-02-15 10:46:51,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51157.93 MB 2025-02-15 10:46:51,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-15 10:46:51,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28672.26 MB 2025-02-15 10:46:51,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19807.08 MB 2025-02-15 10:46:51,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:46:51,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:46:51,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 10:46:51,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:51,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14785.59 MB 2025-02-15 10:46:51,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17799.62 MB 2025-02-15 10:46:51,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:46:51,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-15 10:46:51,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-15 10:46:51,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:46:51,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18100.99 MB 2025-02-15 10:46:51,363 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:46:51,364 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:46:51,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:46:51,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:46:51,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:46:51,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:46:51,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17799.62 MB 2025-02-15 10:46:51,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26238.64 MB 2025-02-15 10:46:51,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:46:51,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-15 10:46:51,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32975.62 MB 2025-02-15 10:46:51,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:46:51,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26238.64 MB 2025-02-15 10:46:51,535 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:46:51,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:51,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:46:51,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:51,537 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:46:51,542 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:46:51,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:46:51,543 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:46:51,543 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:47:07,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:07,819 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:47:07,824 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:47:07,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:07,828 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1956, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:47:07,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:07,829 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1956, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:47:38,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:47:38,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:47:38,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.29 seconds 2025-02-15 10:47:38,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:38,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26598.43 MB 2025-02-15 10:47:38,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33521.13 MB 2025-02-15 10:47:38,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6922.70 MB 2025-02-15 10:47:38,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45560.63 MB 2025-02-15 10:47:38,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40181.43 MB 2025-02-15 10:47:38,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5379.19 MB 2025-02-15 10:47:38,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42411.59 MB 2025-02-15 10:47:38,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:47:38,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:47:38,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:47:38,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:38,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33521.13 MB 2025-02-15 10:47:38,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25946.48 MB 2025-02-15 10:47:38,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7574.64 MB 2025-02-15 10:47:38,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40181.43 MB 2025-02-15 10:47:38,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54773.42 MB 2025-02-15 10:47:38,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14591.98 MB 2025-02-15 10:47:38,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53341.69 MB 2025-02-15 10:47:40,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:47:40,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:47:40,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 10:47:40,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25946.48 MB 2025-02-15 10:47:40,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26477.33 MB 2025-02-15 10:47:40,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:47:40,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54773.42 MB 2025-02-15 10:47:40,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:47:40,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20099.10 MB 2025-02-15 10:47:40,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30455.87 MB 2025-02-15 10:47:40,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:47:40,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:47:40,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:47:40,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26477.33 MB 2025-02-15 10:47:40,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28366.86 MB 2025-02-15 10:47:40,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:47:40,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:47:40,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 10:47:40,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:47:40,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29784.29 MB 2025-02-15 10:47:40,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:47:40,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:47:40,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:47:40,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28366.86 MB 2025-02-15 10:47:40,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30608.72 MB 2025-02-15 10:47:40,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:47:40,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:47:40,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39392.90 MB 2025-02-15 10:47:40,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 10:47:40,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.00 MB 2025-02-15 10:47:40,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:47:40,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:47:40,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:47:40,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26477.33 MB 2025-02-15 10:47:40,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30608.72 MB 2025-02-15 10:47:40,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:47:40,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 10:47:40,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39392.90 MB 2025-02-15 10:47:40,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 10:47:40,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.00 MB 2025-02-15 10:47:40,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:47:40,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:47:40,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:47:40,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32142.26 MB 2025-02-15 10:47:40,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32909.26 MB 2025-02-15 10:47:40,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:47:40,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39392.90 MB 2025-02-15 10:47:40,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-15 10:47:40,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:47:40,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.05 MB 2025-02-15 10:47:40,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:47:40,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:47:40,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:47:40,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33322.15 MB 2025-02-15 10:47:40,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33550.95 MB 2025-02-15 10:47:40,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.80 MB 2025-02-15 10:47:40,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39808.14 MB 2025-02-15 10:47:40,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-15 10:47:40,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:47:40,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33763.52 MB 2025-02-15 10:47:40,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:47:40,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:47:40,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.82 seconds 2025-02-15 10:47:40,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19783.57 MB 2025-02-15 10:47:40,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33752.02 MB 2025-02-15 10:47:40,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13968.46 MB 2025-02-15 10:47:40,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45560.63 MB 2025-02-15 10:47:40,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-15 10:47:40,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5752.49 MB 2025-02-15 10:47:40,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33763.52 MB 2025-02-15 10:47:40,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:47:40,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:47:40,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:47:40,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33752.02 MB 2025-02-15 10:47:40,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.96 MB 2025-02-15 10:47:40,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8964.07 MB 2025-02-15 10:47:40,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39808.14 MB 2025-02-15 10:47:40,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-15 10:47:40,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:47:40,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36263.69 MB 2025-02-15 10:47:40,939 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:47:40,939 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:47:40,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:47:40,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:47:40,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:47:40,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:47:40,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.96 MB 2025-02-15 10:47:40,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33226.98 MB 2025-02-15 10:47:40,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:47:40,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39808.14 MB 2025-02-15 10:47:40,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48198.84 MB 2025-02-15 10:47:40,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:47:40,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33226.98 MB 2025-02-15 10:47:41,107 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:47:41,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:41,108 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:47:41,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:41,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:47:41,114 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:47:41,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:47:41,115 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:47:41,115 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:49:10,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:10,364 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:49:10,369 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:49:10,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:10,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 319, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:49:10,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:10,374 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 319, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:49:15,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:49:15,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:49:15,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.92 seconds 2025-02-15 10:49:15,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:15,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15191.55 MB 2025-02-15 10:49:15,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16320.47 MB 2025-02-15 10:49:15,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1128.92 MB 2025-02-15 10:49:15,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60783.85 MB 2025-02-15 10:49:15,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 10:49:15,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42417.00 MB 2025-02-15 10:49:15,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25198.63 MB 2025-02-15 10:49:15,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:49:15,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:49:15,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:49:15,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:15,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16320.47 MB 2025-02-15 10:49:15,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16060.16 MB 2025-02-15 10:49:15,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -260.31 MB 2025-02-15 10:49:15,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 10:49:15,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20514.34 MB 2025-02-15 10:49:15,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2147.48 MB 2025-02-15 10:49:15,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19186.75 MB 2025-02-15 10:49:16,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:49:16,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:49:16,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-15 10:49:16,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16060.16 MB 2025-02-15 10:49:16,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16330.89 MB 2025-02-15 10:49:16,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.73 MB 2025-02-15 10:49:16,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20514.34 MB 2025-02-15 10:49:16,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19440.60 MB 2025-02-15 10:49:16,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1073.74 MB 2025-02-15 10:49:16,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20315.88 MB 2025-02-15 10:49:16,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:49:16,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:49:16,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:49:16,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16330.89 MB 2025-02-15 10:49:16,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17295.37 MB 2025-02-15 10:49:16,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 964.48 MB 2025-02-15 10:49:16,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19440.60 MB 2025-02-15 10:49:16,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19922.94 MB 2025-02-15 10:49:16,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 482.34 MB 2025-02-15 10:49:16,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18018.26 MB 2025-02-15 10:49:16,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:49:16,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:49:16,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:49:16,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17295.37 MB 2025-02-15 10:49:16,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18438.75 MB 2025-02-15 10:49:16,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1143.38 MB 2025-02-15 10:49:16,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19922.94 MB 2025-02-15 10:49:16,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22817.01 MB 2025-02-15 10:49:16,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2894.07 MB 2025-02-15 10:49:16,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21271.54 MB 2025-02-15 10:49:16,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:49:16,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:49:16,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 10:49:16,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16330.89 MB 2025-02-15 10:49:16,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18438.75 MB 2025-02-15 10:49:16,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2107.86 MB 2025-02-15 10:49:16,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19440.60 MB 2025-02-15 10:49:16,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22817.01 MB 2025-02-15 10:49:16,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3376.41 MB 2025-02-15 10:49:16,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21271.54 MB 2025-02-15 10:49:16,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:49:16,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:49:16,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:49:16,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19220.86 MB 2025-02-15 10:49:16,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19613.07 MB 2025-02-15 10:49:16,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.22 MB 2025-02-15 10:49:16,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22817.01 MB 2025-02-15 10:49:16,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23026.73 MB 2025-02-15 10:49:16,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 209.72 MB 2025-02-15 10:49:16,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19976.59 MB 2025-02-15 10:49:16,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:49:16,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:49:16,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:49:16,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19823.65 MB 2025-02-15 10:49:16,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20034.27 MB 2025-02-15 10:49:16,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.62 MB 2025-02-15 10:49:16,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23026.73 MB 2025-02-15 10:49:16,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23026.73 MB 2025-02-15 10:49:16,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:49:16,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20078.27 MB 2025-02-15 10:49:16,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:49:16,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:49:16,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.14 seconds 2025-02-15 10:49:16,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14080.13 MB 2025-02-15 10:49:16,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20235.35 MB 2025-02-15 10:49:16,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6155.22 MB 2025-02-15 10:49:16,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60783.85 MB 2025-02-15 10:49:16,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23026.73 MB 2025-02-15 10:49:16,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37757.12 MB 2025-02-15 10:49:16,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20235.35 MB 2025-02-15 10:49:16,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:49:16,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:49:16,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 10:49:16,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15146.03 MB 2025-02-15 10:49:16,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18160.06 MB 2025-02-15 10:49:16,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 10:49:16,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23026.73 MB 2025-02-15 10:49:16,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23026.73 MB 2025-02-15 10:49:16,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:49:16,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.43 MB 2025-02-15 10:49:16,799 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:49:16,799 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:49:16,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:49:16,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:49:16,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:49:16,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:49:16,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18160.06 MB 2025-02-15 10:49:16,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26599.08 MB 2025-02-15 10:49:16,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:49:16,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23026.73 MB 2025-02-15 10:49:16,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33516.68 MB 2025-02-15 10:49:16,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 10:49:16,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26599.08 MB 2025-02-15 10:49:16,966 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:49:16,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:16,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:49:16,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:16,969 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:49:16,974 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:49:16,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:16,976 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:49:16,976 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:49:33,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:33,819 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:49:33,824 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:49:33,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:33,828 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2064, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:49:33,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:49:33,829 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2064, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:50:05,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:50:05,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:50:05,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.94 seconds 2025-02-15 10:50:05,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:05,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27350.99 MB 2025-02-15 10:50:05,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34655.37 MB 2025-02-15 10:50:05,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7304.38 MB 2025-02-15 10:50:05,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46101.69 MB 2025-02-15 10:50:05,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40561.02 MB 2025-02-15 10:50:05,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5540.68 MB 2025-02-15 10:50:05,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43617.13 MB 2025-02-15 10:50:05,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:50:05,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:50:05,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:50:05,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:05,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34655.37 MB 2025-02-15 10:50:05,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26508.99 MB 2025-02-15 10:50:05,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8146.38 MB 2025-02-15 10:50:05,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40561.02 MB 2025-02-15 10:50:05,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55912.17 MB 2025-02-15 10:50:05,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15351.15 MB 2025-02-15 10:50:05,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54549.89 MB 2025-02-15 10:50:07,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:50:07,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:50:07,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 10:50:07,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:07,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26508.99 MB 2025-02-15 10:50:07,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27039.83 MB 2025-02-15 10:50:07,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:50:07,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55912.17 MB 2025-02-15 10:50:07,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31186.75 MB 2025-02-15 10:50:07,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24725.42 MB 2025-02-15 10:50:07,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31019.42 MB 2025-02-15 10:50:07,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:50:07,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:50:07,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:50:07,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:07,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27039.83 MB 2025-02-15 10:50:07,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28929.37 MB 2025-02-15 10:50:07,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:50:07,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31186.75 MB 2025-02-15 10:50:07,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32130.47 MB 2025-02-15 10:50:07,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:50:07,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30346.80 MB 2025-02-15 10:50:08,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:50:08,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:50:08,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:50:08,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28929.37 MB 2025-02-15 10:50:08,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31171.22 MB 2025-02-15 10:50:08,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:50:08,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32130.47 MB 2025-02-15 10:50:08,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38736.49 MB 2025-02-15 10:50:08,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:50:08,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36715.50 MB 2025-02-15 10:50:08,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:50:08,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:50:08,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:50:08,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27039.83 MB 2025-02-15 10:50:08,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31171.22 MB 2025-02-15 10:50:08,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:50:08,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31186.75 MB 2025-02-15 10:50:08,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38736.49 MB 2025-02-15 10:50:08,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 10:50:08,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36715.50 MB 2025-02-15 10:50:08,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:50:08,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:50:08,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:50:08,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32704.77 MB 2025-02-15 10:50:08,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33471.77 MB 2025-02-15 10:50:08,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:50:08,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38736.49 MB 2025-02-15 10:50:08,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39153.83 MB 2025-02-15 10:50:08,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 10:50:08,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34179.56 MB 2025-02-15 10:50:08,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:50:08,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:50:08,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:50:08,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33884.66 MB 2025-02-15 10:50:08,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34113.33 MB 2025-02-15 10:50:08,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.68 MB 2025-02-15 10:50:08,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39153.83 MB 2025-02-15 10:50:08,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39153.83 MB 2025-02-15 10:50:08,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:50:08,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34316.80 MB 2025-02-15 10:50:08,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:50:08,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:50:08,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.47 seconds 2025-02-15 10:50:08,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20159.85 MB 2025-02-15 10:50:08,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34314.36 MB 2025-02-15 10:50:08,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14154.51 MB 2025-02-15 10:50:08,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46101.69 MB 2025-02-15 10:50:08,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39153.83 MB 2025-02-15 10:50:08,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6947.86 MB 2025-02-15 10:50:08,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34316.80 MB 2025-02-15 10:50:08,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:50:08,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:50:08,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:50:08,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34314.36 MB 2025-02-15 10:50:08,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25163.47 MB 2025-02-15 10:50:08,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9150.88 MB 2025-02-15 10:50:08,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39153.83 MB 2025-02-15 10:50:08,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39153.83 MB 2025-02-15 10:50:08,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:50:08,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36825.41 MB 2025-02-15 10:50:08,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 10:50:08,591 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:50:08,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:50:08,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:50:08,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:50:08,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:08,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25163.47 MB 2025-02-15 10:50:08,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33600.95 MB 2025-02-15 10:50:08,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 10:50:08,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39153.83 MB 2025-02-15 10:50:08,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47542.44 MB 2025-02-15 10:50:08,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 10:50:08,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33600.95 MB 2025-02-15 10:50:08,759 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 10:50:08,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:08,761 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:50:08,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:08,762 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:50:08,766 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:50:08,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:08,767 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:50:08,767 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:50:17,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:17,200 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:50:17,205 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:50:17,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:17,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 414, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:50:17,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:17,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 414, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:50:23,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:50:23,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:50:23,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.47 seconds 2025-02-15 10:50:23,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:23,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15853.52 MB 2025-02-15 10:50:23,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17318.65 MB 2025-02-15 10:50:23,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1465.12 MB 2025-02-15 10:50:23,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55931.04 MB 2025-02-15 10:50:23,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19308.48 MB 2025-02-15 10:50:23,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36622.57 MB 2025-02-15 10:50:23,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26230.87 MB 2025-02-15 10:50:23,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:50:23,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:50:23,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 10:50:23,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:23,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17318.65 MB 2025-02-15 10:50:23,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17621.62 MB 2025-02-15 10:50:23,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.97 MB 2025-02-15 10:50:23,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19308.48 MB 2025-02-15 10:50:23,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23685.23 MB 2025-02-15 10:50:23,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4376.76 MB 2025-02-15 10:50:23,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22320.09 MB 2025-02-15 10:50:25,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:50:25,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:50:25,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.73 seconds 2025-02-15 10:50:25,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17621.62 MB 2025-02-15 10:50:25,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18094.07 MB 2025-02-15 10:50:25,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.45 MB 2025-02-15 10:50:25,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23685.23 MB 2025-02-15 10:50:25,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-15 10:50:25,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3447.72 MB 2025-02-15 10:50:25,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22047.11 MB 2025-02-15 10:50:25,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:50:25,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:50:25,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:50:25,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18094.07 MB 2025-02-15 10:50:25,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19777.95 MB 2025-02-15 10:50:25,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1683.88 MB 2025-02-15 10:50:25,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-15 10:50:25,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22760.39 MB 2025-02-15 10:50:25,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2522.87 MB 2025-02-15 10:50:25,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21040.51 MB 2025-02-15 10:50:25,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:50:25,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:50:25,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:50:25,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19777.95 MB 2025-02-15 10:50:25,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21774.26 MB 2025-02-15 10:50:25,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1996.31 MB 2025-02-15 10:50:25,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22760.39 MB 2025-02-15 10:50:25,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28441.58 MB 2025-02-15 10:50:25,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5681.18 MB 2025-02-15 10:50:25,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26712.86 MB 2025-02-15 10:50:25,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:50:25,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:50:25,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 10:50:25,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18094.07 MB 2025-02-15 10:50:25,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21774.26 MB 2025-02-15 10:50:25,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3680.19 MB 2025-02-15 10:50:25,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-15 10:50:25,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28441.58 MB 2025-02-15 10:50:25,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8204.06 MB 2025-02-15 10:50:25,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26712.86 MB 2025-02-15 10:50:25,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:50:25,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:50:25,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 10:50:25,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23139.11 MB 2025-02-15 10:50:25,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23821.74 MB 2025-02-15 10:50:25,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 682.63 MB 2025-02-15 10:50:25,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28441.58 MB 2025-02-15 10:50:25,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-15 10:50:25,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 371.20 MB 2025-02-15 10:50:25,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24451.68 MB 2025-02-15 10:50:25,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:50:25,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:50:25,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:50:25,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24189.22 MB 2025-02-15 10:50:25,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24402.65 MB 2025-02-15 10:50:25,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.44 MB 2025-02-15 10:50:25,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-15 10:50:25,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-15 10:50:25,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:50:25,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24550.37 MB 2025-02-15 10:50:25,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:50:25,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:50:25,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.61 seconds 2025-02-15 10:50:25,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:25,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14411.12 MB 2025-02-15 10:50:25,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24603.73 MB 2025-02-15 10:50:25,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10192.61 MB 2025-02-15 10:50:25,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55931.04 MB 2025-02-15 10:50:25,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-15 10:50:25,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27118.27 MB 2025-02-15 10:50:25,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24603.73 MB 2025-02-15 10:50:26,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:50:26,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:50:26,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:50:26,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:26,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24603.73 MB 2025-02-15 10:50:26,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19207.33 MB 2025-02-15 10:50:26,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5396.39 MB 2025-02-15 10:50:26,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-15 10:50:26,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-15 10:50:26,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:50:26,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27818.66 MB 2025-02-15 10:50:26,105 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:50:26,105 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:50:26,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:50:26,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:50:26,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:50:26,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:50:26,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19207.33 MB 2025-02-15 10:50:26,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27646.36 MB 2025-02-15 10:50:26,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:50:26,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-15 10:50:26,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37203.48 MB 2025-02-15 10:50:26,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:50:26,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27646.36 MB 2025-02-15 10:50:26,268 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:50:26,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:26,270 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:50:26,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:26,271 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:50:26,275 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:50:26,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:50:26,276 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:50:26,276 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 10:51:29,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:29,664 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:51:29,669 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:51:29,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:29,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:51:29,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:29,674 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:51:32,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:51:32,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:51:32,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.76 seconds 2025-02-15 10:51:32,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:32,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-15 10:51:32,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-15 10:51:32,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-15 10:51:32,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49788.49 MB 2025-02-15 10:51:32,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18163.43 MB 2025-02-15 10:51:32,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31625.05 MB 2025-02-15 10:51:32,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-15 10:51:32,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:51:32,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:51:32,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:51:32,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:32,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-15 10:51:32,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15037.00 MB 2025-02-15 10:51:32,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.52 MB 2025-02-15 10:51:32,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18163.43 MB 2025-02-15 10:51:32,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18738.05 MB 2025-02-15 10:51:32,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 574.62 MB 2025-02-15 10:51:32,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17125.00 MB 2025-02-15 10:51:33,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:51:33,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:51:33,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 10:51:33,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15037.00 MB 2025-02-15 10:51:33,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15251.99 MB 2025-02-15 10:51:33,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-15 10:51:33,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18738.05 MB 2025-02-15 10:51:33,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19140.71 MB 2025-02-15 10:51:33,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 10:51:33,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19207.69 MB 2025-02-15 10:51:33,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:51:33,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:51:33,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 10:51:33,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-15 10:51:33,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16017.00 MB 2025-02-15 10:51:33,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-15 10:51:33,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19140.71 MB 2025-02-15 10:51:33,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19140.71 MB 2025-02-15 10:51:33,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:51:33,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16591.07 MB 2025-02-15 10:51:33,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:51:33,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:51:33,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:51:33,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16017.00 MB 2025-02-15 10:51:33,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-15 10:51:33,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-15 10:51:33,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19140.71 MB 2025-02-15 10:51:33,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20675.82 MB 2025-02-15 10:51:33,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1535.12 MB 2025-02-15 10:51:33,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-15 10:51:33,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:51:33,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:51:33,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 10:51:33,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-15 10:51:33,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-15 10:51:33,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-15 10:51:33,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19140.71 MB 2025-02-15 10:51:33,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20675.82 MB 2025-02-15 10:51:33,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1535.12 MB 2025-02-15 10:51:33,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-15 10:51:33,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:51:33,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:51:33,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 10:51:33,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.08 MB 2025-02-15 10:51:33,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17857.70 MB 2025-02-15 10:51:33,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.62 MB 2025-02-15 10:51:33,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20675.82 MB 2025-02-15 10:51:33,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-15 10:51:33,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 10:51:33,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18151.99 MB 2025-02-15 10:51:33,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:51:33,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:51:33,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:51:33,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18024.93 MB 2025-02-15 10:51:33,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18229.05 MB 2025-02-15 10:51:33,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.12 MB 2025-02-15 10:51:33,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-15 10:51:33,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 10:51:33,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 10:51:33,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18249.27 MB 2025-02-15 10:51:33,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:51:33,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:51:33,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.73 seconds 2025-02-15 10:51:33,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-15 10:51:33,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18429.80 MB 2025-02-15 10:51:33,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4837.44 MB 2025-02-15 10:51:33,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49788.49 MB 2025-02-15 10:51:33,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 10:51:33,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28942.79 MB 2025-02-15 10:51:33,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18429.80 MB 2025-02-15 10:51:33,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:51:33,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:51:33,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 10:51:33,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18429.80 MB 2025-02-15 10:51:33,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17469.59 MB 2025-02-15 10:51:33,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -960.22 MB 2025-02-15 10:51:33,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20845.69 MB 2025-02-15 10:51:33,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 10:51:33,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:51:33,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19131.95 MB 2025-02-15 10:51:33,686 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 10:51:33,686 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:51:33,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:51:33,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:51:33,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:51:33,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:51:33,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17469.59 MB 2025-02-15 10:51:33,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25895.77 MB 2025-02-15 10:51:33,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 10:51:33,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20845.69 MB 2025-02-15 10:51:33,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29221.72 MB 2025-02-15 10:51:33,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 10:51:33,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25895.77 MB 2025-02-15 10:51:33,850 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 10:51:33,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:33,851 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:51:33,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:33,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:51:33,857 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:51:33,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:51:33,858 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:51:33,858 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:52:52,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:52:52,340 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:52:52,348 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:52:52,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:52:52,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1656, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:52:52,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:52:52,357 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1656, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:53:17,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:53:17,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:53:17,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.50 seconds 2025-02-15 10:53:17,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:17,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24507.98 MB 2025-02-15 10:53:17,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30369.52 MB 2025-02-15 10:53:17,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5861.54 MB 2025-02-15 10:53:17,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37597.74 MB 2025-02-15 10:53:17,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39095.11 MB 2025-02-15 10:53:17,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1497.37 MB 2025-02-15 10:53:17,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39188.68 MB 2025-02-15 10:53:17,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:53:17,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:53:17,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 10:53:17,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:17,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30369.52 MB 2025-02-15 10:53:17,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24386.88 MB 2025-02-15 10:53:17,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5982.64 MB 2025-02-15 10:53:17,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39095.11 MB 2025-02-15 10:53:17,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51615.11 MB 2025-02-15 10:53:17,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12520.00 MB 2025-02-15 10:53:17,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47072.44 MB 2025-02-15 10:53:19,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:53:19,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:53:19,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:53:19,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:19,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24386.88 MB 2025-02-15 10:53:19,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24917.72 MB 2025-02-15 10:53:19,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:53:19,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51615.11 MB 2025-02-15 10:53:19,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34649.15 MB 2025-02-15 10:53:19,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16965.96 MB 2025-02-15 10:53:19,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28896.27 MB 2025-02-15 10:53:19,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:53:19,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:53:19,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:53:19,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:19,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-15 10:53:19,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26807.25 MB 2025-02-15 10:53:19,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:53:19,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 10:53:19,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34649.15 MB 2025-02-15 10:53:19,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:53:19,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28224.68 MB 2025-02-15 10:53:20,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:53:20,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:53:20,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:53:20,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.25 MB 2025-02-15 10:53:20,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-15 10:53:20,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:53:20,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 10:53:20,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 10:53:20,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:53:20,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-15 10:53:20,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:53:20,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:53:20,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 10:53:20,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-15 10:53:20,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-15 10:53:20,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:53:20,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-15 10:53:20,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37008.44 MB 2025-02-15 10:53:20,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 10:53:20,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-15 10:53:20,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:53:20,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:53:20,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:53:20,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30582.65 MB 2025-02-15 10:53:20,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31349.65 MB 2025-02-15 10:53:20,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:53:20,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37008.44 MB 2025-02-15 10:53:20,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-15 10:53:20,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:53:20,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32057.44 MB 2025-02-15 10:53:20,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:53:20,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:53:20,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:53:20,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31762.54 MB 2025-02-15 10:53:20,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31989.30 MB 2025-02-15 10:53:20,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.76 MB 2025-02-15 10:53:20,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37421.58 MB 2025-02-15 10:53:20,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-15 10:53:20,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:53:20,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32178.05 MB 2025-02-15 10:53:20,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:53:20,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:53:20,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.03 seconds 2025-02-15 10:53:20,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18738.34 MB 2025-02-15 10:53:20,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32190.38 MB 2025-02-15 10:53:20,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13452.03 MB 2025-02-15 10:53:20,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37597.74 MB 2025-02-15 10:53:20,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-15 10:53:20,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -176.16 MB 2025-02-15 10:53:20,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32190.38 MB 2025-02-15 10:53:20,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:53:20,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:53:20,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 10:53:20,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32190.38 MB 2025-02-15 10:53:20,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23742.73 MB 2025-02-15 10:53:20,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8447.65 MB 2025-02-15 10:53:20,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37421.58 MB 2025-02-15 10:53:20,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-15 10:53:20,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:53:20,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34702.04 MB 2025-02-15 10:53:20,703 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 10:53:20,704 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:53:20,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:53:20,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:53:20,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:53:20,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:53:20,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23742.73 MB 2025-02-15 10:53:20,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32181.76 MB 2025-02-15 10:53:20,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 10:53:20,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37421.58 MB 2025-02-15 10:53:20,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45812.29 MB 2025-02-15 10:53:20,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 10:53:20,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32181.76 MB 2025-02-15 10:53:20,979 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 10:53:20,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:53:20,982 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:53:20,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:53:20,983 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:53:20,991 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:53:20,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:53:20,993 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:53:20,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:55:03,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:03,570 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:55:03,575 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:55:03,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:03,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1491, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:55:03,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:03,580 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1491, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:55:26,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:55:26,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:55:26,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.94 seconds 2025-02-15 10:55:26,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:26,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23358.23 MB 2025-02-15 10:55:26,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28634.80 MB 2025-02-15 10:55:26,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5276.57 MB 2025-02-15 10:55:26,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58397.29 MB 2025-02-15 10:55:26,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38510.00 MB 2025-02-15 10:55:26,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19887.29 MB 2025-02-15 10:55:26,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37585.95 MB 2025-02-15 10:55:26,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:55:26,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:55:26,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 10:55:26,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:26,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28634.80 MB 2025-02-15 10:55:26,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23529.10 MB 2025-02-15 10:55:26,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5105.70 MB 2025-02-15 10:55:26,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38510.00 MB 2025-02-15 10:55:26,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47831.84 MB 2025-02-15 10:55:26,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9321.84 MB 2025-02-15 10:55:26,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42651.62 MB 2025-02-15 10:55:28,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:55:28,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:55:28,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 10:55:28,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23529.10 MB 2025-02-15 10:55:28,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.94 MB 2025-02-15 10:55:28,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:55:28,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47831.84 MB 2025-02-15 10:55:28,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33231.47 MB 2025-02-15 10:55:28,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14600.37 MB 2025-02-15 10:55:28,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28038.48 MB 2025-02-15 10:55:28,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:55:28,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:55:28,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:55:28,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-15 10:55:28,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25949.47 MB 2025-02-15 10:55:28,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:55:28,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33231.47 MB 2025-02-15 10:55:28,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33231.47 MB 2025-02-15 10:55:28,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:55:28,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27366.90 MB 2025-02-15 10:55:28,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:55:28,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:55:28,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:55:28,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25949.47 MB 2025-02-15 10:55:28,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-15 10:55:28,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:55:28,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33231.47 MB 2025-02-15 10:55:28,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-15 10:55:28,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:55:28,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-15 10:55:28,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:55:28,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:55:28,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:55:28,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-15 10:55:28,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-15 10:55:28,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:55:28,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33231.47 MB 2025-02-15 10:55:28,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-15 10:55:28,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 10:55:28,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-15 10:55:28,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:55:28,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:55:28,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:55:28,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.87 MB 2025-02-15 10:55:28,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30491.87 MB 2025-02-15 10:55:28,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:55:28,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36062.63 MB 2025-02-15 10:55:28,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-15 10:55:28,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:55:28,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.66 MB 2025-02-15 10:55:28,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:55:28,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:55:28,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:55:28,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30904.76 MB 2025-02-15 10:55:28,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31132.10 MB 2025-02-15 10:55:28,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.34 MB 2025-02-15 10:55:28,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-15 10:55:28,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-15 10:55:28,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:55:28,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31348.34 MB 2025-02-15 10:55:28,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:55:28,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:55:28,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.36 seconds 2025-02-15 10:55:28,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:28,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18163.47 MB 2025-02-15 10:55:28,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31332.95 MB 2025-02-15 10:55:28,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13169.48 MB 2025-02-15 10:55:28,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58397.29 MB 2025-02-15 10:55:28,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-15 10:55:28,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21919.43 MB 2025-02-15 10:55:28,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31348.34 MB 2025-02-15 10:55:29,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:55:29,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:55:29,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:55:29,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:29,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31332.95 MB 2025-02-15 10:55:29,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23161.94 MB 2025-02-15 10:55:29,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8171.01 MB 2025-02-15 10:55:29,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-15 10:55:29,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-15 10:55:29,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:55:29,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33839.70 MB 2025-02-15 10:55:29,230 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 10:55:29,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:55:29,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:55:29,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:55:29,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:55:29,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:55:29,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23161.94 MB 2025-02-15 10:55:29,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31584.26 MB 2025-02-15 10:55:29,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 10:55:29,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-15 10:55:29,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44851.79 MB 2025-02-15 10:55:29,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 10:55:29,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31584.26 MB 2025-02-15 10:55:29,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 10:55:29,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:29,396 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:55:29,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:29,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:55:29,402 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:55:29,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:55:29,403 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:55:29,403 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:56:10,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:10,144 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:56:10,149 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:56:10,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:10,153 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:56:10,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:10,154 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:56:43,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:56:43,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:56:43,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.39 seconds 2025-02-15 10:56:43,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:43,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27964.19 MB 2025-02-15 10:56:43,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.04 MB 2025-02-15 10:56:43,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7616.86 MB 2025-02-15 10:56:43,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57411.63 MB 2025-02-15 10:56:43,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40842.04 MB 2025-02-15 10:56:43,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16569.60 MB 2025-02-15 10:56:43,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44456.82 MB 2025-02-15 10:56:43,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:56:43,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:56:43,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 10:56:43,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:43,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35581.04 MB 2025-02-15 10:56:43,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26966.48 MB 2025-02-15 10:56:43,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8614.57 MB 2025-02-15 10:56:43,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40842.04 MB 2025-02-15 10:56:43,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56390.32 MB 2025-02-15 10:56:43,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15548.28 MB 2025-02-15 10:56:43,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55768.19 MB 2025-02-15 10:56:45,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:56:45,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:56:45,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 10:56:45,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:45,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26966.48 MB 2025-02-15 10:56:45,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27497.32 MB 2025-02-15 10:56:45,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:56:45,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56390.32 MB 2025-02-15 10:56:45,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31163.68 MB 2025-02-15 10:56:45,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25226.64 MB 2025-02-15 10:56:45,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31476.90 MB 2025-02-15 10:56:45,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:56:45,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:56:45,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:56:45,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:45,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-15 10:56:45,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29386.85 MB 2025-02-15 10:56:45,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:56:45,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31163.68 MB 2025-02-15 10:56:45,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33051.12 MB 2025-02-15 10:56:45,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:56:45,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30804.28 MB 2025-02-15 10:56:45,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:56:45,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:56:45,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:56:45,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:45,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29386.85 MB 2025-02-15 10:56:45,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-15 10:56:45,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:56:45,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33051.12 MB 2025-02-15 10:56:45,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38713.43 MB 2025-02-15 10:56:45,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 10:56:45,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-15 10:56:45,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:56:45,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:56:45,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:56:45,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:45,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-15 10:56:45,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-15 10:56:45,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:56:45,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31163.68 MB 2025-02-15 10:56:45,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38713.43 MB 2025-02-15 10:56:45,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 10:56:45,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-15 10:56:46,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:56:46,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:56:46,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 10:56:46,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:46,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33162.25 MB 2025-02-15 10:56:46,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33929.25 MB 2025-02-15 10:56:46,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:56:46,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38713.43 MB 2025-02-15 10:56:46,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 10:56:46,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:56:46,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34637.04 MB 2025-02-15 10:56:46,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:56:46,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:56:46,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:56:46,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:46,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34342.14 MB 2025-02-15 10:56:46,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34570.12 MB 2025-02-15 10:56:46,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-15 10:56:46,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 10:56:46,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 10:56:46,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:56:46,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34781.78 MB 2025-02-15 10:56:46,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:56:46,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:56:46,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.95 seconds 2025-02-15 10:56:46,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:46,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20466.45 MB 2025-02-15 10:56:46,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34770.01 MB 2025-02-15 10:56:46,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14303.56 MB 2025-02-15 10:56:46,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57411.63 MB 2025-02-15 10:56:46,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 10:56:46,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18282.97 MB 2025-02-15 10:56:46,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34781.78 MB 2025-02-15 10:56:46,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:56:46,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:56:46,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:56:46,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:46,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34770.01 MB 2025-02-15 10:56:46,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25453.51 MB 2025-02-15 10:56:46,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9316.50 MB 2025-02-15 10:56:46,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 10:56:46,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 10:56:46,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:56:46,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37267.89 MB 2025-02-15 10:56:46,391 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 10:56:46,391 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:56:46,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:56:46,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:56:46,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:56:46,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:56:46,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25453.51 MB 2025-02-15 10:56:46,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33842.65 MB 2025-02-15 10:56:46,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 10:56:46,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 10:56:46,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47471.13 MB 2025-02-15 10:56:46,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 10:56:46,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33842.65 MB 2025-02-15 10:56:46,557 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 10:56:46,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:46,558 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:56:46,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:46,559 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:56:46,564 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:56:46,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:56:46,565 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:56:46,565 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:57:37,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:37,472 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:57:37,477 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:57:37,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:37,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 852, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:57:37,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:37,481 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 852, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:57:50,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:57:50,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:57:50,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.19 seconds 2025-02-15 10:57:50,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:50,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18905.58 MB 2025-02-15 10:57:50,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21921.28 MB 2025-02-15 10:57:50,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3015.70 MB 2025-02-15 10:57:50,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55813.60 MB 2025-02-15 10:57:50,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27837.60 MB 2025-02-15 10:57:50,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27976.01 MB 2025-02-15 10:57:50,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30868.37 MB 2025-02-15 10:57:50,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:57:50,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:57:50,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 10:57:50,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:50,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21921.28 MB 2025-02-15 10:57:50,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20207.13 MB 2025-02-15 10:57:50,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1714.15 MB 2025-02-15 10:57:50,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27837.60 MB 2025-02-15 10:57:50,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35219.57 MB 2025-02-15 10:57:50,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7381.98 MB 2025-02-15 10:57:50,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32109.27 MB 2025-02-15 10:57:52,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:57:52,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:57:52,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:57:52,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:52,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20207.13 MB 2025-02-15 10:57:52,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20737.97 MB 2025-02-15 10:57:52,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:57:52,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35219.57 MB 2025-02-15 10:57:52,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26237.47 MB 2025-02-15 10:57:52,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8982.10 MB 2025-02-15 10:57:52,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24716.52 MB 2025-02-15 10:57:52,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:57:52,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:57:52,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:57:52,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:52,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-15 10:57:52,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22627.51 MB 2025-02-15 10:57:52,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:57:52,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26237.47 MB 2025-02-15 10:57:52,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26237.47 MB 2025-02-15 10:57:52,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:57:52,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24044.94 MB 2025-02-15 10:57:52,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:57:52,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:57:52,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:57:52,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:52,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22627.51 MB 2025-02-15 10:57:52,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-15 10:57:52,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:57:52,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26237.47 MB 2025-02-15 10:57:52,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 10:57:52,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:57:52,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-15 10:57:52,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:57:52,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:57:52,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:57:52,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:52,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-15 10:57:52,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-15 10:57:52,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:57:52,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26237.47 MB 2025-02-15 10:57:52,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-15 10:57:52,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 10:57:52,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-15 10:57:53,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:57:53,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:57:53,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:57:53,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:53,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26402.91 MB 2025-02-15 10:57:53,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27169.91 MB 2025-02-15 10:57:53,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:57:53,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-15 10:57:53,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 10:57:53,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:57:53,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.70 MB 2025-02-15 10:57:53,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:57:53,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:57:53,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:57:53,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:53,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27582.80 MB 2025-02-15 10:57:53,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27811.59 MB 2025-02-15 10:57:53,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 10:57:53,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 10:57:53,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 10:57:53,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:57:53,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28039.63 MB 2025-02-15 10:57:53,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:57:53,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:57:53,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.65 seconds 2025-02-15 10:57:53,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:53,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15937.14 MB 2025-02-15 10:57:53,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28012.29 MB 2025-02-15 10:57:53,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12075.15 MB 2025-02-15 10:57:53,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55813.60 MB 2025-02-15 10:57:53,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 10:57:53,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22554.87 MB 2025-02-15 10:57:53,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28039.63 MB 2025-02-15 10:57:53,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:57:53,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:57:53,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:57:53,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:53,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28012.29 MB 2025-02-15 10:57:53,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20935.97 MB 2025-02-15 10:57:53,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7076.32 MB 2025-02-15 10:57:53,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 10:57:53,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-15 10:57:53,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:57:53,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30519.35 MB 2025-02-15 10:57:53,425 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 10:57:53,425 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:57:53,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:57:53,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:57:53,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:57:53,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:57:53,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20935.97 MB 2025-02-15 10:57:53,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29359.17 MB 2025-02-15 10:57:53,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 10:57:53,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-15 10:57:53,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41634.76 MB 2025-02-15 10:57:53,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 10:57:53,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29359.17 MB 2025-02-15 10:57:53,591 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 10:57:53,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:53,592 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:57:53,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:53,593 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:57:53,598 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:57:53,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:57:53,599 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:57:53,599 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:58:51,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:58:51,662 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:58:51,668 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:58:51,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:58:51,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1276, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:58:51,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:58:51,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1276, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:59:11,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:59:11,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:59:11,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.75 seconds 2025-02-15 10:59:11,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:11,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21860.08 MB 2025-02-15 10:59:11,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26375.77 MB 2025-02-15 10:59:11,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4515.69 MB 2025-02-15 10:59:11,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50010.78 MB 2025-02-15 10:59:11,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37715.18 MB 2025-02-15 10:59:11,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12295.60 MB 2025-02-15 10:59:11,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35181.82 MB 2025-02-15 10:59:11,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:59:11,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:59:11,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 10:59:11,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:11,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26375.77 MB 2025-02-15 10:59:11,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22411.38 MB 2025-02-15 10:59:11,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3964.39 MB 2025-02-15 10:59:11,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37715.18 MB 2025-02-15 10:59:11,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46546.29 MB 2025-02-15 10:59:11,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8831.11 MB 2025-02-15 10:59:11,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39653.67 MB 2025-02-15 10:59:13,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:59:13,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:59:13,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:59:13,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22411.38 MB 2025-02-15 10:59:13,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22942.22 MB 2025-02-15 10:59:13,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:59:13,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46546.29 MB 2025-02-15 10:59:13,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33197.92 MB 2025-02-15 10:59:13,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13348.37 MB 2025-02-15 10:59:13,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26921.80 MB 2025-02-15 10:59:13,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:59:13,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:59:13,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:59:13,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22942.22 MB 2025-02-15 10:59:13,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24831.75 MB 2025-02-15 10:59:13,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:59:13,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33197.92 MB 2025-02-15 10:59:13,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33197.92 MB 2025-02-15 10:59:13,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:59:13,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26249.18 MB 2025-02-15 10:59:13,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:59:13,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:59:13,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:59:13,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24831.75 MB 2025-02-15 10:59:13,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27073.61 MB 2025-02-15 10:59:13,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:59:13,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33197.92 MB 2025-02-15 10:59:13,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-15 10:59:13,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:59:13,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32617.89 MB 2025-02-15 10:59:13,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:59:13,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:59:13,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:59:13,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22942.22 MB 2025-02-15 10:59:13,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27073.61 MB 2025-02-15 10:59:13,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:59:13,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33197.92 MB 2025-02-15 10:59:13,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-15 10:59:13,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 10:59:13,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32617.89 MB 2025-02-15 10:59:13,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:59:13,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:59:13,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:59:13,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28607.15 MB 2025-02-15 10:59:13,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29374.15 MB 2025-02-15 10:59:13,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:59:13,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-15 10:59:13,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-15 10:59:13,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 10:59:13,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30081.94 MB 2025-02-15 10:59:13,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:59:13,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:59:13,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:59:13,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29787.04 MB 2025-02-15 10:59:13,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30015.93 MB 2025-02-15 10:59:13,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-15 10:59:13,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35500.59 MB 2025-02-15 10:59:13,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-15 10:59:13,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:59:13,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30253.37 MB 2025-02-15 10:59:13,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:59:13,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:59:13,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.16 seconds 2025-02-15 10:59:13,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:13,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17414.39 MB 2025-02-15 10:59:13,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30216.78 MB 2025-02-15 10:59:13,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12802.39 MB 2025-02-15 10:59:13,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50010.78 MB 2025-02-15 10:59:13,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-15 10:59:13,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14510.19 MB 2025-02-15 10:59:13,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30253.37 MB 2025-02-15 10:59:14,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:59:14,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:59:14,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:59:14,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:14,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30216.78 MB 2025-02-15 10:59:14,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22414.64 MB 2025-02-15 10:59:14,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7802.14 MB 2025-02-15 10:59:14,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35500.59 MB 2025-02-15 10:59:14,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-15 10:59:14,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:59:14,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32725.07 MB 2025-02-15 10:59:14,121 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 10:59:14,121 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 10:59:14,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:59:14,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:59:14,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:59:14,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:14,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22414.64 MB 2025-02-15 10:59:14,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30841.98 MB 2025-02-15 10:59:14,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 10:59:14,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35500.59 MB 2025-02-15 10:59:14,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43880.81 MB 2025-02-15 10:59:14,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 10:59:14,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30841.98 MB 2025-02-15 10:59:14,288 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 10:59:14,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:14,290 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:59:14,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:14,291 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:59:14,295 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:59:14,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:14,296 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:59:14,296 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 10:59:30,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:30,274 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 10:59:30,279 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 10:59:30,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:30,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1199, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 10:59:30,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:30,283 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1199, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 10:59:48,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 10:59:48,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 10:59:48,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.67 seconds 2025-02-15 10:59:48,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:48,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21323.63 MB 2025-02-15 10:59:48,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25566.82 MB 2025-02-15 10:59:48,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4243.19 MB 2025-02-15 10:59:48,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52261.03 MB 2025-02-15 10:59:48,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29102.18 MB 2025-02-15 10:59:48,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23158.85 MB 2025-02-15 10:59:48,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34418.87 MB 2025-02-15 10:59:49,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 10:59:49,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 10:59:49,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 10:59:49,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:49,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25566.82 MB 2025-02-15 10:59:49,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22012.13 MB 2025-02-15 10:59:49,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3554.69 MB 2025-02-15 10:59:49,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29102.18 MB 2025-02-15 10:59:49,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38897.98 MB 2025-02-15 10:59:49,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9795.80 MB 2025-02-15 10:59:49,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37462.72 MB 2025-02-15 10:59:51,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 10:59:51,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 10:59:51,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 10:59:51,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22012.13 MB 2025-02-15 10:59:51,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22542.97 MB 2025-02-15 10:59:51,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 10:59:51,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38897.98 MB 2025-02-15 10:59:51,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26981.96 MB 2025-02-15 10:59:51,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11916.02 MB 2025-02-15 10:59:51,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26522.55 MB 2025-02-15 10:59:51,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 10:59:51,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 10:59:51,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 10:59:51,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.97 MB 2025-02-15 10:59:51,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24432.50 MB 2025-02-15 10:59:51,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 10:59:51,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26981.96 MB 2025-02-15 10:59:51,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27925.68 MB 2025-02-15 10:59:51,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 10:59:51,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25849.93 MB 2025-02-15 10:59:51,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 10:59:51,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 10:59:51,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 10:59:51,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24432.50 MB 2025-02-15 10:59:51,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26674.36 MB 2025-02-15 10:59:51,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 10:59:51,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27925.68 MB 2025-02-15 10:59:51,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 10:59:51,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 10:59:51,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32218.64 MB 2025-02-15 10:59:51,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 10:59:51,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 10:59:51,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 10:59:51,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.97 MB 2025-02-15 10:59:51,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26674.36 MB 2025-02-15 10:59:51,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 10:59:51,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26981.96 MB 2025-02-15 10:59:51,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34059.85 MB 2025-02-15 10:59:51,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 10:59:51,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32218.64 MB 2025-02-15 10:59:51,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 10:59:51,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 10:59:51,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 10:59:51,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28207.90 MB 2025-02-15 10:59:51,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28974.90 MB 2025-02-15 10:59:51,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 10:59:51,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34059.85 MB 2025-02-15 10:59:51,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-15 10:59:51,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 10:59:51,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29682.69 MB 2025-02-15 10:59:51,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 10:59:51,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 10:59:51,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:59:51,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29387.79 MB 2025-02-15 10:59:51,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29615.81 MB 2025-02-15 10:59:51,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.01 MB 2025-02-15 10:59:51,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34472.98 MB 2025-02-15 10:59:51,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-15 10:59:51,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:59:51,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29817.50 MB 2025-02-15 10:59:51,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 10:59:51,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 10:59:51,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.13 seconds 2025-02-15 10:59:51,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17146.12 MB 2025-02-15 10:59:51,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29816.56 MB 2025-02-15 10:59:51,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12670.44 MB 2025-02-15 10:59:51,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52261.03 MB 2025-02-15 10:59:51,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-15 10:59:51,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17788.04 MB 2025-02-15 10:59:51,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29817.50 MB 2025-02-15 10:59:51,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 10:59:51,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 10:59:51,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 10:59:51,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29816.56 MB 2025-02-15 10:59:51,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22145.65 MB 2025-02-15 10:59:51,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7670.90 MB 2025-02-15 10:59:51,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34472.98 MB 2025-02-15 10:59:51,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-15 10:59:51,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 10:59:51,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32324.23 MB 2025-02-15 10:59:51,702 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 10:59:51,702 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 10:59:51,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 10:59:51,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 10:59:51,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 10:59:51,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 10:59:51,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22145.65 MB 2025-02-15 10:59:51,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30571.83 MB 2025-02-15 10:59:51,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 10:59:51,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34472.98 MB 2025-02-15 10:59:51,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42849.01 MB 2025-02-15 10:59:51,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 10:59:51,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30571.83 MB 2025-02-15 10:59:51,866 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 10:59:51,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:51,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 10:59:51,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:51,868 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 10:59:51,873 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 10:59:51,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 10:59:51,874 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 10:59:51,874 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:00:09,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:09,164 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:00:09,168 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:00:09,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:09,172 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 282, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:00:09,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:09,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 282, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:00:13,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:00:13,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:00:13,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-15 11:00:13,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:13,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.73 MB 2025-02-15 11:00:13,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15931.71 MB 2025-02-15 11:00:13,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.98 MB 2025-02-15 11:00:13,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51225.03 MB 2025-02-15 11:00:13,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20715.67 MB 2025-02-15 11:00:13,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30509.37 MB 2025-02-15 11:00:13,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24858.08 MB 2025-02-15 11:00:13,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:00:13,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:00:13,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:00:13,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:13,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15931.71 MB 2025-02-15 11:00:13,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15417.96 MB 2025-02-15 11:00:13,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -513.75 MB 2025-02-15 11:00:13,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20715.67 MB 2025-02-15 11:00:13,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20715.67 MB 2025-02-15 11:00:13,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:13,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17898.22 MB 2025-02-15 11:00:14,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:00:14,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:00:14,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.68 seconds 2025-02-15 11:00:14,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15417.96 MB 2025-02-15 11:00:14,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15603.75 MB 2025-02-15 11:00:14,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.79 MB 2025-02-15 11:00:14,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20715.67 MB 2025-02-15 11:00:14,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 11:00:14,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 11:00:14,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19587.61 MB 2025-02-15 11:00:14,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:00:14,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:00:14,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:00:14,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15603.69 MB 2025-02-15 11:00:14,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16264.87 MB 2025-02-15 11:00:14,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.18 MB 2025-02-15 11:00:14,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 11:00:14,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19077.79 MB 2025-02-15 11:00:14,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:14,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16760.97 MB 2025-02-15 11:00:14,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:00:14,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:00:14,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:00:14,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.87 MB 2025-02-15 11:00:14,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17049.56 MB 2025-02-15 11:00:14,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.69 MB 2025-02-15 11:00:14,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 11:00:14,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 11:00:14,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 994.05 MB 2025-02-15 11:00:14,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18993.16 MB 2025-02-15 11:00:14,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:00:14,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:00:14,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:00:14,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15603.69 MB 2025-02-15 11:00:14,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17049.56 MB 2025-02-15 11:00:14,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.87 MB 2025-02-15 11:00:14,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19077.79 MB 2025-02-15 11:00:14,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 11:00:14,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 994.05 MB 2025-02-15 11:00:14,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18993.16 MB 2025-02-15 11:00:14,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:00:14,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:00:14,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 11:00:14,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17586.30 MB 2025-02-15 11:00:14,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17854.75 MB 2025-02-15 11:00:14,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.45 MB 2025-02-15 11:00:14,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 11:00:14,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-15 11:00:14,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 140.51 MB 2025-02-15 11:00:14,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18112.63 MB 2025-02-15 11:00:14,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:00:14,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:00:14,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:00:14,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17999.27 MB 2025-02-15 11:00:14,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18211.61 MB 2025-02-15 11:00:14,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.34 MB 2025-02-15 11:00:14,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-15 11:00:14,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-15 11:00:14,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:14,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18217.39 MB 2025-02-15 11:00:14,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:00:14,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:00:14,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.27 seconds 2025-02-15 11:00:14,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13951.22 MB 2025-02-15 11:00:14,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18412.41 MB 2025-02-15 11:00:14,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4461.19 MB 2025-02-15 11:00:14,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51225.03 MB 2025-02-15 11:00:14,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-15 11:00:14,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31012.68 MB 2025-02-15 11:00:14,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18412.41 MB 2025-02-15 11:00:14,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:00:14,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:00:14,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:00:14,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18412.41 MB 2025-02-15 11:00:14,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21422.39 MB 2025-02-15 11:00:14,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.98 MB 2025-02-15 11:00:14,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-15 11:00:14,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23165.14 MB 2025-02-15 11:00:14,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2952.79 MB 2025-02-15 11:00:14,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21723.79 MB 2025-02-15 11:00:14,731 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 11:00:14,732 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:00:14,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:00:14,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:00:14,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:00:14,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:14,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21422.39 MB 2025-02-15 11:00:14,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.72 MB 2025-02-15 11:00:14,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 11:00:14,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23165.14 MB 2025-02-15 11:00:14,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33640.42 MB 2025-02-15 11:00:14,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 11:00:14,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29849.72 MB 2025-02-15 11:00:14,896 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 11:00:14,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:14,897 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:00:14,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:14,898 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:00:14,903 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:00:14,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:14,904 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:00:14,904 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:00:23,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:23,307 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:00:23,314 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:00:23,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:23,320 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 310, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:00:23,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:23,322 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 310, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:00:28,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:00:28,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:00:28,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.91 seconds 2025-02-15 11:00:28,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:28,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24696.66 MB 2025-02-15 11:00:28,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25793.73 MB 2025-02-15 11:00:28,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1097.07 MB 2025-02-15 11:00:28,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42022.73 MB 2025-02-15 11:00:28,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31994.15 MB 2025-02-15 11:00:28,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10028.58 MB 2025-02-15 11:00:28,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34621.01 MB 2025-02-15 11:00:28,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:00:28,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:00:28,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:00:28,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:28,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25793.73 MB 2025-02-15 11:00:28,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26269.80 MB 2025-02-15 11:00:28,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 476.07 MB 2025-02-15 11:00:28,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31994.15 MB 2025-02-15 11:00:28,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33063.70 MB 2025-02-15 11:00:28,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1069.55 MB 2025-02-15 11:00:28,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30036.42 MB 2025-02-15 11:00:29,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:00:29,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:00:29,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.46 seconds 2025-02-15 11:00:29,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:29,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26269.80 MB 2025-02-15 11:00:29,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26671.12 MB 2025-02-15 11:00:29,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 401.33 MB 2025-02-15 11:00:29,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33063.70 MB 2025-02-15 11:00:29,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31425.82 MB 2025-02-15 11:00:29,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 11:00:29,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30609.71 MB 2025-02-15 11:00:29,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:00:29,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:00:29,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:00:29,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:29,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26671.12 MB 2025-02-15 11:00:29,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28097.93 MB 2025-02-15 11:00:29,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1426.81 MB 2025-02-15 11:00:29,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31425.82 MB 2025-02-15 11:00:29,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32140.95 MB 2025-02-15 11:00:29,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 715.13 MB 2025-02-15 11:00:29,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29168.09 MB 2025-02-15 11:00:29,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:00:29,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:00:29,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:00:29,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:29,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28097.93 MB 2025-02-15 11:00:29,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29790.85 MB 2025-02-15 11:00:29,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1692.92 MB 2025-02-15 11:00:29,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32140.95 MB 2025-02-15 11:00:29,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-15 11:00:29,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3921.67 MB 2025-02-15 11:00:29,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33978.87 MB 2025-02-15 11:00:29,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:00:29,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:00:29,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:00:29,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:29,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26671.12 MB 2025-02-15 11:00:29,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29790.85 MB 2025-02-15 11:00:29,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3119.73 MB 2025-02-15 11:00:29,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31425.82 MB 2025-02-15 11:00:29,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-15 11:00:29,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4636.80 MB 2025-02-15 11:00:29,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33978.87 MB 2025-02-15 11:00:30,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:00:30,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:00:30,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:00:30,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:30,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30948.68 MB 2025-02-15 11:00:30,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21960.21 MB 2025-02-15 11:00:30,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8988.47 MB 2025-02-15 11:00:30,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36062.63 MB 2025-02-15 11:00:30,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36230.40 MB 2025-02-15 11:00:30,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 11:00:30,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31151.66 MB 2025-02-15 11:00:30,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:00:30,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:00:30,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:00:30,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:30,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22271.94 MB 2025-02-15 11:00:30,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22492.79 MB 2025-02-15 11:00:30,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.85 MB 2025-02-15 11:00:30,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36230.40 MB 2025-02-15 11:00:30,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36230.40 MB 2025-02-15 11:00:30,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:30,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22596.17 MB 2025-02-15 11:00:30,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:00:30,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:00:30,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.73 seconds 2025-02-15 11:00:30,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:30,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23616.59 MB 2025-02-15 11:00:30,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22693.76 MB 2025-02-15 11:00:30,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -922.83 MB 2025-02-15 11:00:30,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42022.73 MB 2025-02-15 11:00:30,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36230.40 MB 2025-02-15 11:00:30,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5792.33 MB 2025-02-15 11:00:30,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22693.76 MB 2025-02-15 11:00:30,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:00:30,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:00:30,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:00:30,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:30,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22693.76 MB 2025-02-15 11:00:30,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25706.32 MB 2025-02-15 11:00:30,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.56 MB 2025-02-15 11:00:30,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36230.40 MB 2025-02-15 11:00:30,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36230.40 MB 2025-02-15 11:00:30,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:30,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26007.54 MB 2025-02-15 11:00:30,342 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 11:00:30,343 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:00:30,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:00:30,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:00:30,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:00:30,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:30,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18589.41 MB 2025-02-15 11:00:30,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27024.26 MB 2025-02-15 11:00:30,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 11:00:30,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36230.40 MB 2025-02-15 11:00:30,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44616.91 MB 2025-02-15 11:00:30,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 11:00:30,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27024.26 MB 2025-02-15 11:00:30,509 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 11:00:30,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:30,511 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:00:30,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:30,512 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:00:30,516 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:00:30,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:30,517 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:00:30,517 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:00:40,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:40,827 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:00:40,832 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:00:40,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:40,835 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:00:40,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:40,836 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:00:42,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:00:42,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:00:42,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.70 seconds 2025-02-15 11:00:42,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:42,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13714.30 MB 2025-02-15 11:00:42,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14092.97 MB 2025-02-15 11:00:42,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.67 MB 2025-02-15 11:00:42,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57195.63 MB 2025-02-15 11:00:42,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17498.64 MB 2025-02-15 11:00:42,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39696.99 MB 2025-02-15 11:00:42,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22959.18 MB 2025-02-15 11:00:42,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:00:42,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:00:42,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:00:42,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:42,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14092.97 MB 2025-02-15 11:00:42,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14276.43 MB 2025-02-15 11:00:42,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 183.46 MB 2025-02-15 11:00:42,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17498.64 MB 2025-02-15 11:00:42,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17498.64 MB 2025-02-15 11:00:42,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:42,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.49 MB 2025-02-15 11:00:43,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:00:43,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:00:43,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-15 11:00:43,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14276.43 MB 2025-02-15 11:00:43,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14418.43 MB 2025-02-15 11:00:43,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 142.00 MB 2025-02-15 11:00:43,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17498.64 MB 2025-02-15 11:00:43,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17167.29 MB 2025-02-15 11:00:43,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -331.35 MB 2025-02-15 11:00:43,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18362.18 MB 2025-02-15 11:00:43,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:00:43,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:00:43,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:00:43,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-15 11:00:43,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.69 MB 2025-02-15 11:00:43,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 505.33 MB 2025-02-15 11:00:43,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17167.29 MB 2025-02-15 11:00:43,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17167.29 MB 2025-02-15 11:00:43,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:43,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15302.86 MB 2025-02-15 11:00:43,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:00:43,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:00:43,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:00:43,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.69 MB 2025-02-15 11:00:43,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-15 11:00:43,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.77 MB 2025-02-15 11:00:43,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17167.29 MB 2025-02-15 11:00:43,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17674.80 MB 2025-02-15 11:00:43,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 507.51 MB 2025-02-15 11:00:43,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-15 11:00:43,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:00:43,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:00:43,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:00:43,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-15 11:00:43,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-15 11:00:43,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1119.10 MB 2025-02-15 11:00:43,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17167.29 MB 2025-02-15 11:00:43,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17674.80 MB 2025-02-15 11:00:43,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 507.51 MB 2025-02-15 11:00:43,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-15 11:00:43,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:00:43,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:00:43,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 11:00:43,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16130.00 MB 2025-02-15 11:00:43,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16387.77 MB 2025-02-15 11:00:43,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 257.77 MB 2025-02-15 11:00:43,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17674.80 MB 2025-02-15 11:00:43,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17838.37 MB 2025-02-15 11:00:43,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 11:00:43,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16577.10 MB 2025-02-15 11:00:43,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:00:43,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:00:43,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:00:43,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16550.82 MB 2025-02-15 11:00:43,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16779.25 MB 2025-02-15 11:00:43,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.43 MB 2025-02-15 11:00:43,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17838.37 MB 2025-02-15 11:00:43,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17838.37 MB 2025-02-15 11:00:43,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:00:43,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16779.25 MB 2025-02-15 11:00:43,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:00:43,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:00:43,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-15 11:00:43,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13341.50 MB 2025-02-15 11:00:43,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16980.15 MB 2025-02-15 11:00:43,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3638.64 MB 2025-02-15 11:00:43,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57195.63 MB 2025-02-15 11:00:43,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17838.37 MB 2025-02-15 11:00:43,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39357.25 MB 2025-02-15 11:00:43,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16980.15 MB 2025-02-15 11:00:43,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:00:43,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:00:43,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:00:43,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16980.15 MB 2025-02-15 11:00:43,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19991.60 MB 2025-02-15 11:00:43,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.45 MB 2025-02-15 11:00:43,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17838.37 MB 2025-02-15 11:00:43,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21730.69 MB 2025-02-15 11:00:43,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3892.31 MB 2025-02-15 11:00:43,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20293.46 MB 2025-02-15 11:00:43,527 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 11:00:43,527 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:00:43,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:00:43,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:00:43,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:00:43,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:00:43,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19991.60 MB 2025-02-15 11:00:43,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28423.07 MB 2025-02-15 11:00:43,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 11:00:43,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21730.69 MB 2025-02-15 11:00:43,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32212.25 MB 2025-02-15 11:00:43,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 11:00:43,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.07 MB 2025-02-15 11:00:43,699 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 11:00:43,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:43,700 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:00:43,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:43,701 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:00:43,706 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:00:43,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:00:43,707 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:00:43,707 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:01:51,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:51,529 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:01:51,534 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:01:51,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:51,538 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:01:51,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:51,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:01:54,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:01:54,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:01:54,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-15 11:01:54,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:54,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14278.72 MB 2025-02-15 11:01:54,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14944.04 MB 2025-02-15 11:01:54,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 665.32 MB 2025-02-15 11:01:54,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40598.77 MB 2025-02-15 11:01:54,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19266.54 MB 2025-02-15 11:01:54,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21332.23 MB 2025-02-15 11:01:54,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.09 MB 2025-02-15 11:01:54,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:01:54,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:01:54,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:01:54,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:54,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14944.04 MB 2025-02-15 11:01:54,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15266.39 MB 2025-02-15 11:01:54,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.35 MB 2025-02-15 11:01:54,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19266.54 MB 2025-02-15 11:01:54,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19266.54 MB 2025-02-15 11:01:54,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:01:54,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17584.76 MB 2025-02-15 11:01:55,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:01:55,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:01:55,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.89 seconds 2025-02-15 11:01:55,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15266.39 MB 2025-02-15 11:01:55,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15515.88 MB 2025-02-15 11:01:55,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.50 MB 2025-02-15 11:01:55,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19266.54 MB 2025-02-15 11:01:55,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19287.51 MB 2025-02-15 11:01:55,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20.97 MB 2025-02-15 11:01:55,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19437.08 MB 2025-02-15 11:01:55,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:01:55,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:01:55,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:01:55,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-15 11:01:55,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16403.68 MB 2025-02-15 11:01:55,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 887.87 MB 2025-02-15 11:01:55,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 11:01:55,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19287.51 MB 2025-02-15 11:01:55,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:01:55,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17069.88 MB 2025-02-15 11:01:55,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:01:55,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:01:55,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:01:55,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16403.68 MB 2025-02-15 11:01:55,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-15 11:01:55,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1053.71 MB 2025-02-15 11:01:55,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 11:01:55,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21510.49 MB 2025-02-15 11:01:55,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2222.98 MB 2025-02-15 11:01:55,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20067.36 MB 2025-02-15 11:01:55,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:01:55,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:01:55,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:01:55,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-15 11:01:55,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-15 11:01:55,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1941.57 MB 2025-02-15 11:01:55,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19287.51 MB 2025-02-15 11:01:55,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21510.49 MB 2025-02-15 11:01:55,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2222.98 MB 2025-02-15 11:01:55,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20067.36 MB 2025-02-15 11:01:55,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:01:55,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:01:55,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:01:55,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18178.16 MB 2025-02-15 11:01:55,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18538.65 MB 2025-02-15 11:01:55,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 360.49 MB 2025-02-15 11:01:55,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21510.49 MB 2025-02-15 11:01:55,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21703.43 MB 2025-02-15 11:01:55,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 192.94 MB 2025-02-15 11:01:55,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18876.40 MB 2025-02-15 11:01:55,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:01:55,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:01:55,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:01:55,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18732.71 MB 2025-02-15 11:01:55,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18945.01 MB 2025-02-15 11:01:55,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.30 MB 2025-02-15 11:01:55,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21703.43 MB 2025-02-15 11:01:55,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21703.43 MB 2025-02-15 11:01:55,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:01:55,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.20 MB 2025-02-15 11:01:55,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:01:55,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:01:55,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.99 seconds 2025-02-15 11:01:55,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13623.71 MB 2025-02-15 11:01:55,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19146.08 MB 2025-02-15 11:01:55,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5522.37 MB 2025-02-15 11:01:55,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40598.77 MB 2025-02-15 11:01:55,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21703.43 MB 2025-02-15 11:01:55,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18895.34 MB 2025-02-15 11:01:55,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.08 MB 2025-02-15 11:01:55,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:01:55,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:01:55,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:01:55,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19146.08 MB 2025-02-15 11:01:55,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17627.61 MB 2025-02-15 11:01:55,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1518.47 MB 2025-02-15 11:01:55,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21703.43 MB 2025-02-15 11:01:55,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21703.43 MB 2025-02-15 11:01:55,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:01:55,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.09 MB 2025-02-15 11:01:55,817 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:01:55,817 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-15 11:01:55,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:01:55,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:01:55,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:01:55,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:01:55,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17627.61 MB 2025-02-15 11:01:55,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26066.64 MB 2025-02-15 11:01:55,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:01:55,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21703.43 MB 2025-02-15 11:01:55,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32193.38 MB 2025-02-15 11:01:55,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:01:55,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26066.64 MB 2025-02-15 11:01:55,985 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:01:55,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:55,986 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:01:55,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:55,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:01:55,992 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:01:55,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:01:55,993 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:01:55,993 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-15 11:03:09,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:09,769 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:03:09,776 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:03:09,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:09,782 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1413, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:03:09,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:09,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1413, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:03:31,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:03:31,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:03:31,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.79 seconds 2025-02-15 11:03:31,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:31,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22814.72 MB 2025-02-15 11:03:31,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27815.24 MB 2025-02-15 11:03:31,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5000.53 MB 2025-02-15 11:03:31,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44778.39 MB 2025-02-15 11:03:31,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38262.54 MB 2025-02-15 11:03:31,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6515.85 MB 2025-02-15 11:03:31,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36815.94 MB 2025-02-15 11:03:31,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:03:31,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:03:31,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:03:31,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:31,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27815.24 MB 2025-02-15 11:03:31,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23123.60 MB 2025-02-15 11:03:31,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4691.65 MB 2025-02-15 11:03:31,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38262.54 MB 2025-02-15 11:03:31,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48159.00 MB 2025-02-15 11:03:31,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9896.46 MB 2025-02-15 11:03:31,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42698.87 MB 2025-02-15 11:03:33,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:03:33,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:03:33,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 11:03:33,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23123.60 MB 2025-02-15 11:03:33,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.44 MB 2025-02-15 11:03:33,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:03:33,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48159.00 MB 2025-02-15 11:03:33,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29066.53 MB 2025-02-15 11:03:33,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19092.47 MB 2025-02-15 11:03:33,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27633.47 MB 2025-02-15 11:03:33,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:03:33,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:03:33,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:03:33,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-15 11:03:33,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25543.97 MB 2025-02-15 11:03:33,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:03:33,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 11:03:33,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30010.25 MB 2025-02-15 11:03:33,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:03:33,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26961.40 MB 2025-02-15 11:03:33,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:03:33,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:03:33,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:03:33,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25543.97 MB 2025-02-15 11:03:33,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-15 11:03:33,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:03:33,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30010.25 MB 2025-02-15 11:03:33,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 11:03:33,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:03:33,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-15 11:03:33,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:03:33,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:03:33,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:03:33,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-15 11:03:33,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-15 11:03:33,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:03:33,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 11:03:33,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 11:03:33,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:03:33,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-15 11:03:33,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:03:33,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:03:33,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:03:33,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.37 MB 2025-02-15 11:03:33,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30086.37 MB 2025-02-15 11:03:33,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:03:33,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35672.56 MB 2025-02-15 11:03:33,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:03:33,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 11:03:33,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30794.16 MB 2025-02-15 11:03:33,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:03:33,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:03:33,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:03:33,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30499.26 MB 2025-02-15 11:03:33,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30728.33 MB 2025-02-15 11:03:33,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 11:03:33,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:03:33,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:03:33,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:03:33,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30962.59 MB 2025-02-15 11:03:33,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:03:33,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:03:33,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.20 seconds 2025-02-15 11:03:33,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:33,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17891.71 MB 2025-02-15 11:03:33,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30929.25 MB 2025-02-15 11:03:33,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13037.54 MB 2025-02-15 11:03:33,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44778.39 MB 2025-02-15 11:03:33,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:03:33,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8692.70 MB 2025-02-15 11:03:33,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30962.59 MB 2025-02-15 11:03:34,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:03:34,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:03:34,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:03:34,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:34,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30929.25 MB 2025-02-15 11:03:34,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22893.81 MB 2025-02-15 11:03:34,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8035.44 MB 2025-02-15 11:03:34,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:03:34,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:03:34,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:03:34,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.08 MB 2025-02-15 11:03:34,280 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 11:03:34,280 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:03:34,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:03:34,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:03:34,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:03:34,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:03:34,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22893.81 MB 2025-02-15 11:03:34,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31327.11 MB 2025-02-15 11:03:34,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-15 11:03:34,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:03:34,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44470.11 MB 2025-02-15 11:03:34,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 11:03:34,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31327.11 MB 2025-02-15 11:03:34,446 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 11:03:34,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:34,447 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:03:34,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:34,448 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:03:34,453 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:03:34,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:03:34,454 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:03:34,454 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:04:40,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:04:40,647 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:04:40,652 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:04:40,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:04:40,656 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1572, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:04:40,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:04:40,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1572, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:05:04,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:05:04,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:05:04,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.24 seconds 2025-02-15 11:05:04,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:04,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23922.65 MB 2025-02-15 11:05:04,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29486.40 MB 2025-02-15 11:05:04,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5563.74 MB 2025-02-15 11:05:04,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52854.52 MB 2025-02-15 11:05:04,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38809.89 MB 2025-02-15 11:05:04,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14044.63 MB 2025-02-15 11:05:04,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38376.86 MB 2025-02-15 11:05:05,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:05:05,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:05:05,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:05:05,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:05,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29486.40 MB 2025-02-15 11:05:05,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.19 MB 2025-02-15 11:05:05,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5536.21 MB 2025-02-15 11:05:05,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38809.89 MB 2025-02-15 11:05:05,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49830.43 MB 2025-02-15 11:05:05,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11020.53 MB 2025-02-15 11:05:05,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46060.33 MB 2025-02-15 11:05:06,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:05:06,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:05:06,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:05:06,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:06,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23950.19 MB 2025-02-15 11:05:06,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24481.03 MB 2025-02-15 11:05:06,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:05:06,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49830.43 MB 2025-02-15 11:05:06,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29053.94 MB 2025-02-15 11:05:06,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20776.48 MB 2025-02-15 11:05:06,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28460.62 MB 2025-02-15 11:05:06,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:05:06,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:05:06,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:05:06,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:06,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-15 11:05:06,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.56 MB 2025-02-15 11:05:06,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:05:06,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-15 11:05:06,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29997.66 MB 2025-02-15 11:05:06,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:05:06,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27787.99 MB 2025-02-15 11:05:07,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:05:07,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:05:07,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:05:07,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26370.56 MB 2025-02-15 11:05:07,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28613.47 MB 2025-02-15 11:05:07,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 11:05:07,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29997.66 MB 2025-02-15 11:05:07,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36368.81 MB 2025-02-15 11:05:07,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 11:05:07,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34157.75 MB 2025-02-15 11:05:07,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:05:07,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:05:07,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:05:07,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-15 11:05:07,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28613.47 MB 2025-02-15 11:05:07,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 11:05:07,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-15 11:05:07,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36368.81 MB 2025-02-15 11:05:07,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:05:07,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34157.75 MB 2025-02-15 11:05:07,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:05:07,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:05:07,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:05:07,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30147.01 MB 2025-02-15 11:05:07,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30914.01 MB 2025-02-15 11:05:07,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:05:07,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36368.81 MB 2025-02-15 11:05:07,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:05:07,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:05:07,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31621.80 MB 2025-02-15 11:05:07,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:05:07,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:05:07,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:05:07,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31326.90 MB 2025-02-15 11:05:07,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31554.78 MB 2025-02-15 11:05:07,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.88 MB 2025-02-15 11:05:07,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:05:07,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:05:07,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:05:07,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31789.76 MB 2025-02-15 11:05:07,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:05:07,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:05:07,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.69 seconds 2025-02-15 11:05:07,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18445.68 MB 2025-02-15 11:05:07,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31755.76 MB 2025-02-15 11:05:07,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13310.08 MB 2025-02-15 11:05:07,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52854.52 MB 2025-02-15 11:05:07,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:05:07,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16068.38 MB 2025-02-15 11:05:07,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31789.76 MB 2025-02-15 11:05:07,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:05:07,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:05:07,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:05:07,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31755.76 MB 2025-02-15 11:05:07,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23448.55 MB 2025-02-15 11:05:07,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8307.21 MB 2025-02-15 11:05:07,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:05:07,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:05:07,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:05:07,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34266.19 MB 2025-02-15 11:05:07,638 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 11:05:07,639 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:05:07,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:05:07,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:05:07,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:05:07,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:05:07,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23448.55 MB 2025-02-15 11:05:07,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31883.40 MB 2025-02-15 11:05:07,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 11:05:07,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:05:07,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40980.45 MB 2025-02-15 11:05:07,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 11:05:07,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31883.40 MB 2025-02-15 11:05:07,803 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 11:05:07,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:05:07,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:05:07,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:05:07,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:05:07,810 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:05:07,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:05:07,811 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:05:07,811 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:06:05,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:05,997 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:06:06,002 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:06:06,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:06,006 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1520, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:06:06,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:06,007 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1520, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:06:29,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:06:29,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:06:29,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.43 seconds 2025-02-15 11:06:29,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:29,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23560.31 MB 2025-02-15 11:06:29,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28939.50 MB 2025-02-15 11:06:29,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5379.19 MB 2025-02-15 11:06:29,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53559.16 MB 2025-02-15 11:06:29,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38625.35 MB 2025-02-15 11:06:29,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14933.82 MB 2025-02-15 11:06:29,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37788.02 MB 2025-02-15 11:06:29,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:06:29,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:06:29,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:06:29,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:29,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.50 MB 2025-02-15 11:06:29,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23679.86 MB 2025-02-15 11:06:29,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5259.65 MB 2025-02-15 11:06:29,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38625.35 MB 2025-02-15 11:06:29,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47645.20 MB 2025-02-15 11:06:29,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9019.85 MB 2025-02-15 11:06:29,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42502.84 MB 2025-02-15 11:06:31,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:06:31,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:06:31,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:06:31,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23679.86 MB 2025-02-15 11:06:31,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24210.70 MB 2025-02-15 11:06:31,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:06:31,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47645.20 MB 2025-02-15 11:06:31,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-15 11:06:31,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14399.05 MB 2025-02-15 11:06:31,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28189.25 MB 2025-02-15 11:06:31,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:06:31,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:06:31,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:06:31,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-15 11:06:31,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.23 MB 2025-02-15 11:06:31,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:06:31,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-15 11:06:31,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-15 11:06:31,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:06:31,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27517.66 MB 2025-02-15 11:06:31,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:06:31,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:06:31,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:06:31,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26100.23 MB 2025-02-15 11:06:31,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-15 11:06:31,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:06:31,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-15 11:06:31,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-15 11:06:31,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:06:31,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-15 11:06:31,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:06:31,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:06:31,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:06:31,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-15 11:06:31,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-15 11:06:31,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:06:31,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-15 11:06:31,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-15 11:06:31,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:06:31,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-15 11:06:31,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:06:31,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:06:31,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:06:31,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29875.63 MB 2025-02-15 11:06:31,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30642.63 MB 2025-02-15 11:06:31,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:06:31,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36077.31 MB 2025-02-15 11:06:31,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36494.64 MB 2025-02-15 11:06:31,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:06:31,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31350.42 MB 2025-02-15 11:06:31,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:06:31,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:06:31,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:06:31,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31055.52 MB 2025-02-15 11:06:31,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31283.50 MB 2025-02-15 11:06:31,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-15 11:06:31,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36494.64 MB 2025-02-15 11:06:31,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36494.64 MB 2025-02-15 11:06:31,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:06:31,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31489.38 MB 2025-02-15 11:06:31,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:06:31,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:06:31,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.86 seconds 2025-02-15 11:06:31,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:31,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18264.51 MB 2025-02-15 11:06:31,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31484.35 MB 2025-02-15 11:06:31,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13219.84 MB 2025-02-15 11:06:31,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53559.16 MB 2025-02-15 11:06:31,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36494.64 MB 2025-02-15 11:06:31,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17064.53 MB 2025-02-15 11:06:31,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31489.38 MB 2025-02-15 11:06:32,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:06:32,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:06:32,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:06:32,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:32,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31484.35 MB 2025-02-15 11:06:32,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23251.57 MB 2025-02-15 11:06:32,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8232.78 MB 2025-02-15 11:06:32,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36494.64 MB 2025-02-15 11:06:32,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36494.64 MB 2025-02-15 11:06:32,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:06:32,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33981.27 MB 2025-02-15 11:06:32,153 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 11:06:32,153 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:06:32,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:06:32,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:06:32,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:06:32,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:06:32,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23251.57 MB 2025-02-15 11:06:32,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31640.72 MB 2025-02-15 11:06:32,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 11:06:32,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36494.64 MB 2025-02-15 11:06:32,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40665.87 MB 2025-02-15 11:06:32,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 11:06:32,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31640.72 MB 2025-02-15 11:06:32,319 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 11:06:32,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:32,320 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:06:32,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:32,321 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:06:32,326 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:06:32,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:06:32,327 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:06:32,327 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:07:33,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:33,195 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:07:33,202 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:07:33,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:33,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:07:33,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:33,211 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:07:51,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:07:51,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:07:51,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.46 seconds 2025-02-15 11:07:51,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:51,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21281.72 MB 2025-02-15 11:07:51,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25503.68 MB 2025-02-15 11:07:51,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4221.96 MB 2025-02-15 11:07:51,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49008.35 MB 2025-02-15 11:07:51,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29043.46 MB 2025-02-15 11:07:51,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19964.89 MB 2025-02-15 11:07:51,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34376.97 MB 2025-02-15 11:07:51,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:07:51,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:07:51,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:07:51,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:51,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25503.68 MB 2025-02-15 11:07:51,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21980.93 MB 2025-02-15 11:07:51,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3522.75 MB 2025-02-15 11:07:51,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29043.46 MB 2025-02-15 11:07:51,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39550.19 MB 2025-02-15 11:07:51,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10506.73 MB 2025-02-15 11:07:51,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38108.84 MB 2025-02-15 11:07:53,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:07:53,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:07:53,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:07:53,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:53,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21980.93 MB 2025-02-15 11:07:53,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22511.78 MB 2025-02-15 11:07:53,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:07:53,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39550.19 MB 2025-02-15 11:07:53,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26944.21 MB 2025-02-15 11:07:53,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12605.98 MB 2025-02-15 11:07:53,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26491.36 MB 2025-02-15 11:07:53,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:07:53,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:07:53,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:07:53,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:53,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22511.78 MB 2025-02-15 11:07:53,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24401.31 MB 2025-02-15 11:07:53,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:07:53,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 11:07:53,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-15 11:07:53,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:07:53,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25818.74 MB 2025-02-15 11:07:53,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:07:53,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:07:53,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:07:53,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:53,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24401.31 MB 2025-02-15 11:07:53,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26643.17 MB 2025-02-15 11:07:53,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:07:53,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-15 11:07:53,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 11:07:53,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:07:53,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32187.45 MB 2025-02-15 11:07:53,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:07:53,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:07:53,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:07:53,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:53,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22511.78 MB 2025-02-15 11:07:53,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26643.17 MB 2025-02-15 11:07:53,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:07:53,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 11:07:53,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 11:07:53,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:07:53,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32187.45 MB 2025-02-15 11:07:54,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:07:54,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:07:54,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:07:54,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:54,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28176.71 MB 2025-02-15 11:07:54,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28943.71 MB 2025-02-15 11:07:54,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:07:54,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33550.24 MB 2025-02-15 11:07:54,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 11:07:54,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:07:54,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29651.50 MB 2025-02-15 11:07:54,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:07:54,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:07:54,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:07:54,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:54,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29356.60 MB 2025-02-15 11:07:54,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29585.07 MB 2025-02-15 11:07:54,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-15 11:07:54,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 11:07:54,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 11:07:54,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:07:54,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29823.31 MB 2025-02-15 11:07:54,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:07:54,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:07:54,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.91 seconds 2025-02-15 11:07:54,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:54,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17125.21 MB 2025-02-15 11:07:54,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29785.92 MB 2025-02-15 11:07:54,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12660.71 MB 2025-02-15 11:07:54,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49008.35 MB 2025-02-15 11:07:54,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 11:07:54,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15040.77 MB 2025-02-15 11:07:54,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29823.31 MB 2025-02-15 11:07:54,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:07:54,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:07:54,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:07:54,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:54,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29785.92 MB 2025-02-15 11:07:54,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22119.40 MB 2025-02-15 11:07:54,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7666.52 MB 2025-02-15 11:07:54,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 11:07:54,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 11:07:54,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:07:54,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32288.99 MB 2025-02-15 11:07:54,406 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-15 11:07:54,406 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:07:54,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:07:54,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:07:54,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:07:54,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:07:54,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22119.40 MB 2025-02-15 11:07:54,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30529.21 MB 2025-02-15 11:07:54,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-15 11:07:54,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 11:07:54,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42328.92 MB 2025-02-15 11:07:54,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-15 11:07:54,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30529.21 MB 2025-02-15 11:07:54,571 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-15 11:07:54,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:54,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:07:54,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:54,573 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:07:54,578 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:07:54,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:07:54,579 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:07:54,579 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:09:04,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:04,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:09:04,242 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:09:04,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:04,246 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1264, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:09:04,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:04,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1264, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:09:23,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:09:23,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:09:23,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.54 seconds 2025-02-15 11:09:23,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:23,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21776.46 MB 2025-02-15 11:09:23,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26249.69 MB 2025-02-15 11:09:23,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4473.23 MB 2025-02-15 11:09:23,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54869.88 MB 2025-02-15 11:09:23,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37654.36 MB 2025-02-15 11:09:23,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17215.52 MB 2025-02-15 11:09:23,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35098.20 MB 2025-02-15 11:09:23,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:09:23,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:09:23,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:09:23,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:23,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26249.69 MB 2025-02-15 11:09:23,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22348.99 MB 2025-02-15 11:09:23,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3900.69 MB 2025-02-15 11:09:23,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37654.36 MB 2025-02-15 11:09:23,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46533.71 MB 2025-02-15 11:09:23,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8879.34 MB 2025-02-15 11:09:23,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39601.10 MB 2025-02-15 11:09:25,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:09:25,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:09:25,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:09:25,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:25,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.99 MB 2025-02-15 11:09:25,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.83 MB 2025-02-15 11:09:25,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:09:25,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46533.71 MB 2025-02-15 11:09:25,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33181.14 MB 2025-02-15 11:09:25,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13352.57 MB 2025-02-15 11:09:25,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26858.38 MB 2025-02-15 11:09:25,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:09:25,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:09:25,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:09:25,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:25,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-15 11:09:25,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24769.37 MB 2025-02-15 11:09:25,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:09:25,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33181.14 MB 2025-02-15 11:09:25,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33181.14 MB 2025-02-15 11:09:25,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:09:25,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26186.80 MB 2025-02-15 11:09:26,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:09:26,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:09:26,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:09:26,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24769.37 MB 2025-02-15 11:09:26,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-15 11:09:26,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:09:26,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33181.14 MB 2025-02-15 11:09:26,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-15 11:09:26,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 11:09:26,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-15 11:09:26,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:09:26,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:09:26,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:09:26,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-15 11:09:26,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-15 11:09:26,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:09:26,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33181.14 MB 2025-02-15 11:09:26,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-15 11:09:26,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 11:09:26,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-15 11:09:26,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:09:26,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:09:26,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:09:26,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28544.77 MB 2025-02-15 11:09:26,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29311.77 MB 2025-02-15 11:09:26,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:09:26,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-15 11:09:26,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34068.23 MB 2025-02-15 11:09:26,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:09:26,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30019.56 MB 2025-02-15 11:09:26,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:09:26,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:09:26,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:09:26,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.66 MB 2025-02-15 11:09:26,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29951.77 MB 2025-02-15 11:09:26,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.12 MB 2025-02-15 11:09:26,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34068.23 MB 2025-02-15 11:09:26,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34068.23 MB 2025-02-15 11:09:26,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:09:26,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30183.82 MB 2025-02-15 11:09:26,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:09:26,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:09:26,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.98 seconds 2025-02-15 11:09:26,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17372.58 MB 2025-02-15 11:09:26,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30151.72 MB 2025-02-15 11:09:26,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12779.13 MB 2025-02-15 11:09:26,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54869.88 MB 2025-02-15 11:09:26,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34068.23 MB 2025-02-15 11:09:26,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20801.65 MB 2025-02-15 11:09:26,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30183.82 MB 2025-02-15 11:09:26,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:09:26,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:09:26,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:09:26,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30151.72 MB 2025-02-15 11:09:26,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22360.36 MB 2025-02-15 11:09:26,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7791.36 MB 2025-02-15 11:09:26,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34068.23 MB 2025-02-15 11:09:26,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34068.23 MB 2025-02-15 11:09:26,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:09:26,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32649.25 MB 2025-02-15 11:09:26,516 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 11:09:26,516 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:09:26,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:09:26,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:09:26,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:09:26,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:09:26,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22360.36 MB 2025-02-15 11:09:26,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30752.43 MB 2025-02-15 11:09:26,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.07 MB 2025-02-15 11:09:26,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34068.23 MB 2025-02-15 11:09:26,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38239.47 MB 2025-02-15 11:09:26,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 11:09:26,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30752.43 MB 2025-02-15 11:09:26,737 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 11:09:26,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:26,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:09:26,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:26,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:09:26,744 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:09:26,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:26,745 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:09:26,745 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:09:37,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:37,277 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:09:37,284 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:09:37,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:37,291 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1694, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:09:37,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:09:37,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1694, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:10:03,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:10:03,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:10:03,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.49 seconds 2025-02-15 11:10:03,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:03,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24772.77 MB 2025-02-15 11:10:03,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.53 MB 2025-02-15 11:10:03,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5995.76 MB 2025-02-15 11:10:03,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46581.94 MB 2025-02-15 11:10:03,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39160.12 MB 2025-02-15 11:10:03,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7421.82 MB 2025-02-15 11:10:03,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39679.96 MB 2025-02-15 11:10:03,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:10:03,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:10:03,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:10:03,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:03,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30768.53 MB 2025-02-15 11:10:03,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24584.43 MB 2025-02-15 11:10:03,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6184.10 MB 2025-02-15 11:10:03,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39160.12 MB 2025-02-15 11:10:03,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49673.14 MB 2025-02-15 11:10:03,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10513.02 MB 2025-02-15 11:10:03,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44451.08 MB 2025-02-15 11:10:05,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:10:05,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:10:05,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:10:05,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:05,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24584.43 MB 2025-02-15 11:10:05,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25115.27 MB 2025-02-15 11:10:05,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:10:05,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49673.14 MB 2025-02-15 11:10:05,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34579.94 MB 2025-02-15 11:10:05,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15093.20 MB 2025-02-15 11:10:05,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29093.82 MB 2025-02-15 11:10:05,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:10:05,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:10:05,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:10:05,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:05,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25115.27 MB 2025-02-15 11:10:05,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27004.80 MB 2025-02-15 11:10:05,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:10:05,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34579.94 MB 2025-02-15 11:10:05,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34579.94 MB 2025-02-15 11:10:05,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:10:05,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28422.23 MB 2025-02-15 11:10:06,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:10:06,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:10:06,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:10:06,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27004.80 MB 2025-02-15 11:10:06,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29246.66 MB 2025-02-15 11:10:06,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:10:06,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34579.94 MB 2025-02-15 11:10:06,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37411.09 MB 2025-02-15 11:10:06,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:10:06,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34790.94 MB 2025-02-15 11:10:06,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:10:06,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:10:06,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 11:10:06,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25115.27 MB 2025-02-15 11:10:06,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29246.66 MB 2025-02-15 11:10:06,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:10:06,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34579.94 MB 2025-02-15 11:10:06,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37411.09 MB 2025-02-15 11:10:06,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:10:06,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34790.94 MB 2025-02-15 11:10:06,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:10:06,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:10:06,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:10:06,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30780.20 MB 2025-02-15 11:10:06,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31547.20 MB 2025-02-15 11:10:06,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:10:06,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37411.09 MB 2025-02-15 11:10:06,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37826.33 MB 2025-02-15 11:10:06,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:10:06,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32254.99 MB 2025-02-15 11:10:06,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:10:06,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:10:06,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:10:06,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31960.09 MB 2025-02-15 11:10:06,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32188.27 MB 2025-02-15 11:10:06,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-15 11:10:06,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37826.33 MB 2025-02-15 11:10:06,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37826.33 MB 2025-02-15 11:10:06,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:10:06,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32419.13 MB 2025-02-15 11:10:06,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:10:06,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:10:06,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.96 seconds 2025-02-15 11:10:06,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18870.74 MB 2025-02-15 11:10:06,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32389.12 MB 2025-02-15 11:10:06,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13518.38 MB 2025-02-15 11:10:06,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46581.94 MB 2025-02-15 11:10:06,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37826.33 MB 2025-02-15 11:10:06,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8755.61 MB 2025-02-15 11:10:06,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32419.13 MB 2025-02-15 11:10:06,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:10:06,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:10:06,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:10:06,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32389.12 MB 2025-02-15 11:10:06,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23864.93 MB 2025-02-15 11:10:06,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8524.19 MB 2025-02-15 11:10:06,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37826.33 MB 2025-02-15 11:10:06,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37826.33 MB 2025-02-15 11:10:06,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:10:06,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34892.18 MB 2025-02-15 11:10:06,549 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-15 11:10:06,550 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:10:06,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:10:06,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:10:06,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:10:06,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:10:06,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23864.93 MB 2025-02-15 11:10:06,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32274.73 MB 2025-02-15 11:10:06,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-15 11:10:06,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37826.33 MB 2025-02-15 11:10:06,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46187.68 MB 2025-02-15 11:10:06,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-15 11:10:06,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32274.73 MB 2025-02-15 11:10:06,713 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-15 11:10:06,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:10:06,715 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:10:06,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:10:06,716 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:10:06,720 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:10:06,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:10:06,722 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:10:06,722 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:11:10,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:10,299 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:11:10,304 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:11:10,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:10,309 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 160, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:11:10,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:10,310 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 160, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:11:12,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:11:12,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:11:12,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-15 11:11:12,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:12,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14083.61 MB 2025-02-15 11:11:12,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14649.84 MB 2025-02-15 11:11:12,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 566.23 MB 2025-02-15 11:11:12,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58728.64 MB 2025-02-15 11:11:12,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17423.14 MB 2025-02-15 11:11:12,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41305.51 MB 2025-02-15 11:11:12,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23554.98 MB 2025-02-15 11:11:12,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:11:12,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:11:12,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:11:12,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:12,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14649.84 MB 2025-02-15 11:11:12,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14924.57 MB 2025-02-15 11:11:12,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 274.73 MB 2025-02-15 11:11:12,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17423.14 MB 2025-02-15 11:11:12,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18264.10 MB 2025-02-15 11:11:12,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 840.96 MB 2025-02-15 11:11:12,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16897.66 MB 2025-02-15 11:11:13,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:11:13,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:11:13,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 11:11:13,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14924.57 MB 2025-02-15 11:11:13,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15136.91 MB 2025-02-15 11:11:13,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.34 MB 2025-02-15 11:11:13,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18264.10 MB 2025-02-15 11:11:13,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17792.24 MB 2025-02-15 11:11:13,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 11:11:13,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19095.26 MB 2025-02-15 11:11:13,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:11:13,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:11:13,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:11:13,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.84 MB 2025-02-15 11:11:13,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15892.47 MB 2025-02-15 11:11:13,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 755.63 MB 2025-02-15 11:11:13,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17792.24 MB 2025-02-15 11:11:13,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17792.24 MB 2025-02-15 11:11:13,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:11:13,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16459.45 MB 2025-02-15 11:11:13,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:11:13,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:11:13,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:11:13,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15892.47 MB 2025-02-15 11:11:13,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16789.25 MB 2025-02-15 11:11:13,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 896.78 MB 2025-02-15 11:11:13,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17792.24 MB 2025-02-15 11:11:13,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 11:11:13,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 11:11:13,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.93 MB 2025-02-15 11:11:13,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:11:13,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:11:13,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:11:13,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.84 MB 2025-02-15 11:11:13,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16789.25 MB 2025-02-15 11:11:13,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1652.41 MB 2025-02-15 11:11:13,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17792.24 MB 2025-02-15 11:11:13,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19868.42 MB 2025-02-15 11:11:13,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 11:11:13,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.93 MB 2025-02-15 11:11:13,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:11:13,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:11:13,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:11:13,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17402.67 MB 2025-02-15 11:11:13,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17709.47 MB 2025-02-15 11:11:13,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.80 MB 2025-02-15 11:11:13,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19868.42 MB 2025-02-15 11:11:13,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20032.00 MB 2025-02-15 11:11:13,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 11:11:13,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18000.67 MB 2025-02-15 11:11:13,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:11:13,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:11:13,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:11:13,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17874.63 MB 2025-02-15 11:11:13,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18102.70 MB 2025-02-15 11:11:13,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.07 MB 2025-02-15 11:11:13,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20032.00 MB 2025-02-15 11:11:13,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20032.00 MB 2025-02-15 11:11:13,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:11:13,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18127.33 MB 2025-02-15 11:11:13,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:11:13,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:11:13,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-15 11:11:13,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:13,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13526.16 MB 2025-02-15 11:11:13,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18303.33 MB 2025-02-15 11:11:13,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4777.17 MB 2025-02-15 11:11:13,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58728.64 MB 2025-02-15 11:11:13,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20032.00 MB 2025-02-15 11:11:13,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38696.65 MB 2025-02-15 11:11:13,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18303.33 MB 2025-02-15 11:11:14,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:11:14,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:11:14,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:11:14,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:14,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.33 MB 2025-02-15 11:11:14,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.06 MB 2025-02-15 11:11:14,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -912.27 MB 2025-02-15 11:11:14,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20032.00 MB 2025-02-15 11:11:14,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20032.00 MB 2025-02-15 11:11:14,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:11:14,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19105.30 MB 2025-02-15 11:11:14,073 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 11:11:14,073 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:11:14,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:11:14,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:11:14,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:11:14,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:14,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.06 MB 2025-02-15 11:11:14,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25811.84 MB 2025-02-15 11:11:14,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 11:11:14,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20032.00 MB 2025-02-15 11:11:14,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30496.78 MB 2025-02-15 11:11:14,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 11:11:14,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25811.84 MB 2025-02-15 11:11:14,240 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 11:11:14,242 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:14,242 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:11:14,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:14,243 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:11:14,247 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:11:14,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:14,248 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:11:14,249 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:11:30,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:30,049 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:11:30,054 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:11:30,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:30,058 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1462, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:11:30,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:30,059 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1462, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:11:52,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:11:52,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:11:52,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.64 seconds 2025-02-15 11:11:52,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:52,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23156.16 MB 2025-02-15 11:11:52,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28330.09 MB 2025-02-15 11:11:52,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5173.94 MB 2025-02-15 11:11:52,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-15 11:11:52,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38396.76 MB 2025-02-15 11:11:52,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 11:11:52,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37157.38 MB 2025-02-15 11:11:52,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:11:52,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:11:52,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:11:52,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:52,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28330.09 MB 2025-02-15 11:11:52,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23378.33 MB 2025-02-15 11:11:52,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4951.76 MB 2025-02-15 11:11:52,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38396.76 MB 2025-02-15 11:11:52,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46609.20 MB 2025-02-15 11:11:52,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8212.45 MB 2025-02-15 11:11:52,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40765.95 MB 2025-02-15 11:11:54,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:11:54,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:11:54,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:11:54,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:54,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23378.33 MB 2025-02-15 11:11:54,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23909.17 MB 2025-02-15 11:11:54,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:11:54,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46609.20 MB 2025-02-15 11:11:54,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29035.07 MB 2025-02-15 11:11:54,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17574.13 MB 2025-02-15 11:11:54,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27887.72 MB 2025-02-15 11:11:54,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:11:54,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:11:54,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:11:54,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:54,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23909.17 MB 2025-02-15 11:11:54,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25798.71 MB 2025-02-15 11:11:54,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:11:54,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29035.07 MB 2025-02-15 11:11:54,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29978.79 MB 2025-02-15 11:11:54,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:11:54,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27216.14 MB 2025-02-15 11:11:54,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:11:54,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:11:54,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 11:11:54,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:54,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25798.71 MB 2025-02-15 11:11:54,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28040.56 MB 2025-02-15 11:11:54,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:11:54,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29978.79 MB 2025-02-15 11:11:54,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35641.10 MB 2025-02-15 11:11:54,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:11:54,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33584.85 MB 2025-02-15 11:11:54,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:11:54,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:11:54,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 11:11:54,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:54,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23909.17 MB 2025-02-15 11:11:54,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28040.56 MB 2025-02-15 11:11:54,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:11:54,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29035.07 MB 2025-02-15 11:11:54,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35641.10 MB 2025-02-15 11:11:54,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:11:54,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33584.85 MB 2025-02-15 11:11:55,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:11:55,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:11:55,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 11:11:55,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:55,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29574.11 MB 2025-02-15 11:11:55,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.11 MB 2025-02-15 11:11:55,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:11:55,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35641.10 MB 2025-02-15 11:11:55,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:11:55,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:11:55,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31048.90 MB 2025-02-15 11:11:55,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:11:55,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:11:55,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 11:11:55,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:55,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30754.00 MB 2025-02-15 11:11:55,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30982.34 MB 2025-02-15 11:11:55,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-15 11:11:55,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:11:55,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:11:55,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:11:55,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.79 MB 2025-02-15 11:11:55,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:11:55,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:11:55,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.23 seconds 2025-02-15 11:11:55,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:55,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18062.43 MB 2025-02-15 11:11:55,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.24 MB 2025-02-15 11:11:55,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13120.81 MB 2025-02-15 11:11:55,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-15 11:11:55,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:11:55,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2810.18 MB 2025-02-15 11:11:55,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.79 MB 2025-02-15 11:11:55,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:11:55,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:11:55,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 11:11:55,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:55,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31183.24 MB 2025-02-15 11:11:55,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23064.15 MB 2025-02-15 11:11:55,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8119.09 MB 2025-02-15 11:11:55,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:11:55,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:11:55,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:11:55,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33692.76 MB 2025-02-15 11:11:55,604 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 11:11:55,604 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:11:55,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:11:55,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:11:55,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:11:55,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:11:55,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23064.15 MB 2025-02-15 11:11:55,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31495.62 MB 2025-02-15 11:11:55,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 11:11:55,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:11:55,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44442.85 MB 2025-02-15 11:11:55,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 11:11:55,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31495.62 MB 2025-02-15 11:11:55,880 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 11:11:55,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:55,883 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:11:55,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:55,884 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:11:55,892 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:11:55,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:11:55,894 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:11:55,895 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:13:00,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:00,384 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:13:00,389 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:13:00,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:00,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:13:00,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:00,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:13:05,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:13:05,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:13:05,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.72 seconds 2025-02-15 11:13:05,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:05,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15107.93 MB 2025-02-15 11:13:05,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16194.39 MB 2025-02-15 11:13:05,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1086.46 MB 2025-02-15 11:13:05,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52827.26 MB 2025-02-15 11:13:05,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20948.45 MB 2025-02-15 11:13:05,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31878.81 MB 2025-02-15 11:13:05,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25032.29 MB 2025-02-15 11:13:05,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:13:05,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:13:05,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:13:05,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:05,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16194.39 MB 2025-02-15 11:13:05,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16713.68 MB 2025-02-15 11:13:05,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 519.30 MB 2025-02-15 11:13:05,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20948.45 MB 2025-02-15 11:13:05,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23643.29 MB 2025-02-15 11:13:05,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2694.84 MB 2025-02-15 11:13:05,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20492.48 MB 2025-02-15 11:13:06,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:13:06,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:13:06,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.46 seconds 2025-02-15 11:13:06,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16713.68 MB 2025-02-15 11:13:06,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17119.78 MB 2025-02-15 11:13:06,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.09 MB 2025-02-15 11:13:06,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23643.29 MB 2025-02-15 11:13:06,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-15 11:13:06,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2722.10 MB 2025-02-15 11:13:06,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21054.24 MB 2025-02-15 11:13:06,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:13:06,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:13:06,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:13:06,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-15 11:13:06,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18564.98 MB 2025-02-15 11:13:06,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.20 MB 2025-02-15 11:13:06,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-15 11:13:06,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22368.22 MB 2025-02-15 11:13:06,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1447.03 MB 2025-02-15 11:13:06,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.31 MB 2025-02-15 11:13:06,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:13:06,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:13:06,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:13:06,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18564.98 MB 2025-02-15 11:13:06,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.01 MB 2025-02-15 11:13:06,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1715.04 MB 2025-02-15 11:13:06,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22368.22 MB 2025-02-15 11:13:06,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26709.33 MB 2025-02-15 11:13:06,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4341.10 MB 2025-02-15 11:13:06,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24521.37 MB 2025-02-15 11:13:06,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:13:06,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:13:06,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:13:06,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-15 11:13:06,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.01 MB 2025-02-15 11:13:06,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3160.24 MB 2025-02-15 11:13:06,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-15 11:13:06,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26709.33 MB 2025-02-15 11:13:06,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5788.14 MB 2025-02-15 11:13:06,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24521.37 MB 2025-02-15 11:13:06,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:13:06,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:13:06,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:13:06,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21453.17 MB 2025-02-15 11:13:06,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22039.93 MB 2025-02-15 11:13:06,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 586.76 MB 2025-02-15 11:13:06,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-15 11:13:06,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27026.00 MB 2025-02-15 11:13:06,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 316.67 MB 2025-02-15 11:13:06,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22581.39 MB 2025-02-15 11:13:06,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:13:06,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:13:06,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:13:06,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22355.79 MB 2025-02-15 11:13:06,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22562.15 MB 2025-02-15 11:13:06,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.35 MB 2025-02-15 11:13:06,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27026.00 MB 2025-02-15 11:13:06,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27030.19 MB 2025-02-15 11:13:06,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 11:13:06,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22699.99 MB 2025-02-15 11:13:06,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:13:06,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:13:06,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.53 seconds 2025-02-15 11:13:06,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:06,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14038.32 MB 2025-02-15 11:13:06,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22763.22 MB 2025-02-15 11:13:06,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8724.90 MB 2025-02-15 11:13:06,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52827.26 MB 2025-02-15 11:13:06,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27030.19 MB 2025-02-15 11:13:06,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25797.07 MB 2025-02-15 11:13:06,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22763.22 MB 2025-02-15 11:13:07,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:13:07,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:13:07,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:13:07,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:07,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22763.22 MB 2025-02-15 11:13:07,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25777.25 MB 2025-02-15 11:13:07,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 11:13:07,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27030.19 MB 2025-02-15 11:13:07,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27030.19 MB 2025-02-15 11:13:07,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:13:07,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26078.62 MB 2025-02-15 11:13:07,213 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:13:07,213 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:13:07,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:13:07,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:13:07,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:13:07,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:13:07,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.57 MB 2025-02-15 11:13:07,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27037.59 MB 2025-02-15 11:13:07,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:13:07,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27030.19 MB 2025-02-15 11:13:07,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37520.15 MB 2025-02-15 11:13:07,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:13:07,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27037.59 MB 2025-02-15 11:13:07,379 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:13:07,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:07,380 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:13:07,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:07,381 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:13:07,386 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:13:07,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:07,387 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:13:07,387 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:13:51,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:51,638 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:13:51,643 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:13:51,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:51,647 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1445, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:13:51,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:13:51,648 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1445, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:14:13,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:14:13,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:14:13,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.26 seconds 2025-02-15 11:14:13,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:13,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23037.70 MB 2025-02-15 11:14:13,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28151.47 MB 2025-02-15 11:14:13,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5113.77 MB 2025-02-15 11:14:13,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50105.16 MB 2025-02-15 11:14:13,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38375.78 MB 2025-02-15 11:14:13,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11729.37 MB 2025-02-15 11:14:13,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37038.92 MB 2025-02-15 11:14:14,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:14:14,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:14:14,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:14:14,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:14,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28151.47 MB 2025-02-15 11:14:14,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23289.96 MB 2025-02-15 11:14:14,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4861.52 MB 2025-02-15 11:14:14,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38375.78 MB 2025-02-15 11:14:14,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47999.61 MB 2025-02-15 11:14:14,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9623.83 MB 2025-02-15 11:14:14,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42622.68 MB 2025-02-15 11:14:15,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:14:15,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:14:15,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:14:15,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:15,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23289.96 MB 2025-02-15 11:14:15,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23820.80 MB 2025-02-15 11:14:15,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:14:15,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47999.61 MB 2025-02-15 11:14:15,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29066.53 MB 2025-02-15 11:14:15,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18933.09 MB 2025-02-15 11:14:15,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27799.34 MB 2025-02-15 11:14:15,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:14:15,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:14:15,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:14:15,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:15,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-15 11:14:15,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25710.33 MB 2025-02-15 11:14:15,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:14:15,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 11:14:15,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30010.25 MB 2025-02-15 11:14:15,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:14:15,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27127.76 MB 2025-02-15 11:14:16,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:14:16,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:14:16,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:14:16,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25710.33 MB 2025-02-15 11:14:16,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-15 11:14:16,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:14:16,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30010.25 MB 2025-02-15 11:14:16,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 11:14:16,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:14:16,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-15 11:14:16,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:14:16,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:14:16,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:14:16,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-15 11:14:16,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-15 11:14:16,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:14:16,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29066.53 MB 2025-02-15 11:14:16,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35672.56 MB 2025-02-15 11:14:16,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:14:16,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-15 11:14:16,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:14:16,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:14:16,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:14:16,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29485.73 MB 2025-02-15 11:14:16,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30252.73 MB 2025-02-15 11:14:16,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:14:16,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35672.56 MB 2025-02-15 11:14:16,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:14:16,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 11:14:16,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30960.52 MB 2025-02-15 11:14:16,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:14:16,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:14:16,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:14:16,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30665.62 MB 2025-02-15 11:14:16,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30895.06 MB 2025-02-15 11:14:16,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.44 MB 2025-02-15 11:14:16,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:14:16,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:14:16,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:16,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31101.07 MB 2025-02-15 11:14:16,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:14:16,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:14:16,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.71 seconds 2025-02-15 11:14:16,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18003.20 MB 2025-02-15 11:14:16,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31096.13 MB 2025-02-15 11:14:16,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13092.93 MB 2025-02-15 11:14:16,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50105.16 MB 2025-02-15 11:14:16,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:14:16,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14019.46 MB 2025-02-15 11:14:16,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31101.07 MB 2025-02-15 11:14:16,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:14:16,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:14:16,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:14:16,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31096.13 MB 2025-02-15 11:14:16,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.59 MB 2025-02-15 11:14:16,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8088.54 MB 2025-02-15 11:14:16,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:14:16,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36085.69 MB 2025-02-15 11:14:16,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:16,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33607.80 MB 2025-02-15 11:14:16,647 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:14:16,647 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:14:16,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:14:16,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:14:16,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:14:16,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:16,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.59 MB 2025-02-15 11:14:16,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31446.61 MB 2025-02-15 11:14:16,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:14:16,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36085.69 MB 2025-02-15 11:14:16,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44476.40 MB 2025-02-15 11:14:16,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:14:16,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31446.61 MB 2025-02-15 11:14:16,815 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:14:16,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:16,816 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:14:16,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:16,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:14:16,822 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:14:16,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:16,823 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:14:16,823 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:14:22,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:22,915 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:14:22,923 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:14:22,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:22,930 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:14:22,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:22,932 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:14:41,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:14:41,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:14:41,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.66 seconds 2025-02-15 11:14:41,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:41,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.94 MB 2025-02-15 11:14:41,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25430.13 MB 2025-02-15 11:14:41,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-15 11:14:41,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57061.41 MB 2025-02-15 11:14:41,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29068.62 MB 2025-02-15 11:14:41,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27992.78 MB 2025-02-15 11:14:41,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.19 MB 2025-02-15 11:14:41,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:14:41,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:14:41,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:14:41,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:41,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25430.13 MB 2025-02-15 11:14:41,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21944.54 MB 2025-02-15 11:14:41,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3485.59 MB 2025-02-15 11:14:41,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29068.62 MB 2025-02-15 11:14:41,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39489.37 MB 2025-02-15 11:14:41,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10420.75 MB 2025-02-15 11:14:41,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37951.18 MB 2025-02-15 11:14:43,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:14:43,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:14:43,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 11:14:43,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:43,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21944.54 MB 2025-02-15 11:14:43,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22475.38 MB 2025-02-15 11:14:43,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:14:43,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39489.37 MB 2025-02-15 11:14:43,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26994.54 MB 2025-02-15 11:14:43,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12494.83 MB 2025-02-15 11:14:43,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26456.01 MB 2025-02-15 11:14:43,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:14:43,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:14:43,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:14:43,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:43,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22475.38 MB 2025-02-15 11:14:43,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24364.92 MB 2025-02-15 11:14:43,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:14:43,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26994.54 MB 2025-02-15 11:14:43,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27938.26 MB 2025-02-15 11:14:43,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:14:43,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25782.35 MB 2025-02-15 11:14:43,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:14:43,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:14:43,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:14:43,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:43,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24364.92 MB 2025-02-15 11:14:43,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26607.82 MB 2025-02-15 11:14:43,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 11:14:43,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27938.26 MB 2025-02-15 11:14:43,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34309.41 MB 2025-02-15 11:14:43,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 11:14:43,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32152.10 MB 2025-02-15 11:14:43,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:14:43,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:14:43,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:14:43,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:43,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22475.38 MB 2025-02-15 11:14:43,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26607.82 MB 2025-02-15 11:14:43,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 11:14:43,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26994.54 MB 2025-02-15 11:14:43,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34309.41 MB 2025-02-15 11:14:43,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:14:43,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32152.10 MB 2025-02-15 11:14:44,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:14:44,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:14:44,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:14:44,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:44,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28141.37 MB 2025-02-15 11:14:44,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28908.37 MB 2025-02-15 11:14:44,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:14:44,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34309.41 MB 2025-02-15 11:14:44,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:14:44,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:14:44,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29616.16 MB 2025-02-15 11:14:44,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:14:44,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:14:44,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:14:44,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:44,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29321.26 MB 2025-02-15 11:14:44,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29548.94 MB 2025-02-15 11:14:44,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.68 MB 2025-02-15 11:14:44,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 11:14:44,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:14:44,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:44,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.60 MB 2025-02-15 11:14:44,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:14:44,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:14:44,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.15 seconds 2025-02-15 11:14:44,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:44,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17100.83 MB 2025-02-15 11:14:44,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29749.79 MB 2025-02-15 11:14:44,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12648.97 MB 2025-02-15 11:14:44,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57061.41 MB 2025-02-15 11:14:44,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:14:44,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22336.77 MB 2025-02-15 11:14:44,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.60 MB 2025-02-15 11:14:44,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:14:44,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:14:44,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:14:44,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:44,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29749.79 MB 2025-02-15 11:14:44,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22095.73 MB 2025-02-15 11:14:44,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7654.06 MB 2025-02-15 11:14:44,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 11:14:44,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:14:44,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:44,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32253.47 MB 2025-02-15 11:14:44,373 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 11:14:44,373 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:14:44,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:14:44,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:14:44,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:14:44,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:44,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22095.73 MB 2025-02-15 11:14:44,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30508.16 MB 2025-02-15 11:14:44,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-15 11:14:44,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 11:14:44,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43088.08 MB 2025-02-15 11:14:44,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 11:14:44,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30508.16 MB 2025-02-15 11:14:44,544 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 11:14:44,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:44,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:14:44,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:44,546 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:14:44,551 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:14:44,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:44,552 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:14:44,552 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:14:54,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:54,782 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:14:54,788 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 295 2025-02-15 11:14:54,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:54,792 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 116, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:14:54,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:54,793 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 116, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:14:56,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:14:56,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:14:56,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.83 seconds 2025-02-15 11:14:56,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:56,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13777.01 MB 2025-02-15 11:14:56,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14187.53 MB 2025-02-15 11:14:56,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.52 MB 2025-02-15 11:14:56,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51451.53 MB 2025-02-15 11:14:56,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:56,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33317.45 MB 2025-02-15 11:14:56,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23021.89 MB 2025-02-15 11:14:56,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:14:56,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:14:56,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:14:56,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:56,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14187.53 MB 2025-02-15 11:14:56,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14386.43 MB 2025-02-15 11:14:56,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.89 MB 2025-02-15 11:14:56,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:56,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:56,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:56,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15002.27 MB 2025-02-15 11:14:57,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:14:57,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:14:57,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.56 seconds 2025-02-15 11:14:57,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14386.43 MB 2025-02-15 11:14:57,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14540.37 MB 2025-02-15 11:14:57,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 153.94 MB 2025-02-15 11:14:57,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:57,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:57,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18472.89 MB 2025-02-15 11:14:57,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:14:57,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:14:57,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:14:57,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.30 MB 2025-02-15 11:14:57,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15088.14 MB 2025-02-15 11:14:57,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 547.83 MB 2025-02-15 11:14:57,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:57,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:57,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15499.20 MB 2025-02-15 11:14:57,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:14:57,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:14:57,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:14:57,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15088.14 MB 2025-02-15 11:14:57,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15753.52 MB 2025-02-15 11:14:57,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 665.39 MB 2025-02-15 11:14:57,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:57,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:57,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.12 MB 2025-02-15 11:14:57,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:14:57,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:14:57,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:14:57,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.30 MB 2025-02-15 11:14:57,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15753.52 MB 2025-02-15 11:14:57,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1213.22 MB 2025-02-15 11:14:57,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:57,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18134.07 MB 2025-02-15 11:14:57,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.12 MB 2025-02-15 11:14:57,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:14:57,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:14:57,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 11:14:57,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16395.91 MB 2025-02-15 11:14:57,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17139.18 MB 2025-02-15 11:14:57,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 743.27 MB 2025-02-15 11:14:57,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18134.07 MB 2025-02-15 11:14:57,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18249.42 MB 2025-02-15 11:14:57,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 115.34 MB 2025-02-15 11:14:57,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17344.44 MB 2025-02-15 11:14:57,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:14:57,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:14:57,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:14:57,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17464.19 MB 2025-02-15 11:14:57,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17557.32 MB 2025-02-15 11:14:57,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 93.14 MB 2025-02-15 11:14:57,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18249.42 MB 2025-02-15 11:14:57,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18249.42 MB 2025-02-15 11:14:57,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17557.32 MB 2025-02-15 11:14:57,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:14:57,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:14:57,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-15 11:14:57,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13372.86 MB 2025-02-15 11:14:57,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17641.94 MB 2025-02-15 11:14:57,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4269.08 MB 2025-02-15 11:14:57,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51451.53 MB 2025-02-15 11:14:57,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18249.42 MB 2025-02-15 11:14:57,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33202.11 MB 2025-02-15 11:14:57,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17641.94 MB 2025-02-15 11:14:57,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:14:57,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:14:57,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:14:57,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.37 MB 2025-02-15 11:14:57,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15407.88 MB 2025-02-15 11:14:57,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1268.50 MB 2025-02-15 11:14:57,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18249.42 MB 2025-02-15 11:14:57,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18249.42 MB 2025-02-15 11:14:57,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:14:57,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15534.71 MB 2025-02-15 11:14:57,484 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3427, cut from 3429 2025-02-15 11:14:57,484 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:14:57,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:14:57,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:14:57,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:14:57,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:14:57,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15407.88 MB 2025-02-15 11:14:57,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18960.62 MB 2025-02-15 11:14:57,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3552.75 MB 2025-02-15 11:14:57,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18249.42 MB 2025-02-15 11:14:57,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22663.92 MB 2025-02-15 11:14:57,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4414.50 MB 2025-02-15 11:14:57,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18960.62 MB 2025-02-15 11:14:57,554 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3148] 2025-02-15 11:14:57,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:57,556 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 308, 128256]), torch.float32, cuda:0] 2025-02-15 11:14:57,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:57,557 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 309]), torch.int64, cuda:0] 2025-02-15 11:14:57,563 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [296, 308] 2025-02-15 11:14:57,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:14:57,564 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:14:57,564 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:15:24,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:24,323 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:15:24,328 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:15:24,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:24,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 168, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:15:24,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:24,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 168, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:15:26,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:15:26,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:15:26,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.61 seconds 2025-02-15 11:15:26,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:26,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.36 MB 2025-02-15 11:15:26,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14733.90 MB 2025-02-15 11:15:26,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.54 MB 2025-02-15 11:15:26,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26195.53 MB 2025-02-15 11:15:26,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19472.06 MB 2025-02-15 11:15:26,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6723.47 MB 2025-02-15 11:15:26,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23610.73 MB 2025-02-15 11:15:26,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:15:26,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:15:26,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:15:26,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:26,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14733.90 MB 2025-02-15 11:15:26,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14895.54 MB 2025-02-15 11:15:26,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 161.64 MB 2025-02-15 11:15:26,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19472.06 MB 2025-02-15 11:15:26,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19472.06 MB 2025-02-15 11:15:26,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:15:26,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.87 MB 2025-02-15 11:15:27,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:15:27,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:15:27,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 11:15:27,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14895.54 MB 2025-02-15 11:15:27,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15094.61 MB 2025-02-15 11:15:27,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.07 MB 2025-02-15 11:15:27,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19472.06 MB 2025-02-15 11:15:27,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19472.06 MB 2025-02-15 11:15:27,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:15:27,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19066.23 MB 2025-02-15 11:15:27,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:15:27,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:15:27,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:15:27,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15094.54 MB 2025-02-15 11:15:27,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15802.94 MB 2025-02-15 11:15:27,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 708.40 MB 2025-02-15 11:15:27,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19472.06 MB 2025-02-15 11:15:27,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19472.06 MB 2025-02-15 11:15:27,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:15:27,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16334.48 MB 2025-02-15 11:15:27,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:15:27,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:15:27,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:15:27,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15802.94 MB 2025-02-15 11:15:27,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16643.68 MB 2025-02-15 11:15:27,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.74 MB 2025-02-15 11:15:27,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19472.06 MB 2025-02-15 11:15:27,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19826.48 MB 2025-02-15 11:15:27,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 354.42 MB 2025-02-15 11:15:27,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18723.27 MB 2025-02-15 11:15:27,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:15:27,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:15:27,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:15:27,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15094.54 MB 2025-02-15 11:15:27,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16643.68 MB 2025-02-15 11:15:27,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1549.14 MB 2025-02-15 11:15:27,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19472.06 MB 2025-02-15 11:15:27,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19826.48 MB 2025-02-15 11:15:27,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 354.42 MB 2025-02-15 11:15:27,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18723.27 MB 2025-02-15 11:15:27,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:15:27,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:15:27,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:15:27,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17218.76 MB 2025-02-15 11:15:27,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17506.38 MB 2025-02-15 11:15:27,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 287.63 MB 2025-02-15 11:15:27,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19826.48 MB 2025-02-15 11:15:27,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19979.57 MB 2025-02-15 11:15:27,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 11:15:27,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17781.95 MB 2025-02-15 11:15:27,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:15:27,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:15:27,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:15:27,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17661.22 MB 2025-02-15 11:15:27,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17880.21 MB 2025-02-15 11:15:27,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.99 MB 2025-02-15 11:15:27,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19979.57 MB 2025-02-15 11:15:27,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19979.57 MB 2025-02-15 11:15:27,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:15:27,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17888.86 MB 2025-02-15 11:15:27,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:15:27,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:15:27,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.52 seconds 2025-02-15 11:15:27,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:27,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13554.03 MB 2025-02-15 11:15:27,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18081.04 MB 2025-02-15 11:15:27,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4527.01 MB 2025-02-15 11:15:27,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26195.53 MB 2025-02-15 11:15:27,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19979.57 MB 2025-02-15 11:15:27,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6215.96 MB 2025-02-15 11:15:27,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18081.04 MB 2025-02-15 11:15:28,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:15:28,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:15:28,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:15:28,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:28,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18081.04 MB 2025-02-15 11:15:28,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17374.34 MB 2025-02-15 11:15:28,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -706.70 MB 2025-02-15 11:15:28,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19979.57 MB 2025-02-15 11:15:28,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19979.57 MB 2025-02-15 11:15:28,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:15:28,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19084.48 MB 2025-02-15 11:15:28,147 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 11:15:28,148 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:15:28,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:15:28,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:15:28,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:15:28,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:15:28,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17374.34 MB 2025-02-15 11:15:28,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25803.46 MB 2025-02-15 11:15:28,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 11:15:28,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19979.57 MB 2025-02-15 11:15:28,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30454.84 MB 2025-02-15 11:15:28,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 11:15:28,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25803.46 MB 2025-02-15 11:15:28,314 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 11:15:28,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:28,316 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:15:28,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:28,317 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:15:28,321 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:15:28,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:15:28,323 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:15:28,323 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:16:24,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:24,794 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:16:24,802 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:16:24,809 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:24,809 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 534, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:16:24,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:24,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 534, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:16:33,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:16:33,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:16:33,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.23 seconds 2025-02-15 11:16:33,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:33,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16689.70 MB 2025-02-15 11:16:33,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18579.50 MB 2025-02-15 11:16:33,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.80 MB 2025-02-15 11:16:33,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38835.06 MB 2025-02-15 11:16:33,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22101.88 MB 2025-02-15 11:16:33,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16733.18 MB 2025-02-15 11:16:33,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27520.84 MB 2025-02-15 11:16:33,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:16:33,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:16:33,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 11:16:33,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:33,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18579.50 MB 2025-02-15 11:16:33,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18553.95 MB 2025-02-15 11:16:33,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -25.55 MB 2025-02-15 11:16:33,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22101.88 MB 2025-02-15 11:16:33,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27026.00 MB 2025-02-15 11:16:33,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4924.11 MB 2025-02-15 11:16:33,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26308.82 MB 2025-02-15 11:16:35,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:16:35,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:16:35,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:16:35,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18553.95 MB 2025-02-15 11:16:35,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19084.79 MB 2025-02-15 11:16:35,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:16:35,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27026.00 MB 2025-02-15 11:16:35,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21625.83 MB 2025-02-15 11:16:35,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5400.17 MB 2025-02-15 11:16:35,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23064.38 MB 2025-02-15 11:16:35,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:16:35,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:16:35,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:16:35,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19084.79 MB 2025-02-15 11:16:35,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20974.33 MB 2025-02-15 11:16:35,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:16:35,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21625.83 MB 2025-02-15 11:16:35,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24456.99 MB 2025-02-15 11:16:35,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:16:35,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22391.75 MB 2025-02-15 11:16:35,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:16:35,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:16:35,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:16:35,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20974.33 MB 2025-02-15 11:16:35,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23216.18 MB 2025-02-15 11:16:35,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:16:35,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24456.99 MB 2025-02-15 11:16:35,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30828.13 MB 2025-02-15 11:16:35,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 11:16:35,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28761.51 MB 2025-02-15 11:16:35,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:16:35,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:16:35,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:16:35,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19084.79 MB 2025-02-15 11:16:35,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23216.18 MB 2025-02-15 11:16:35,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:16:35,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21625.83 MB 2025-02-15 11:16:35,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30828.13 MB 2025-02-15 11:16:35,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-15 11:16:35,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28761.51 MB 2025-02-15 11:16:35,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:16:35,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:16:35,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:16:35,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24750.77 MB 2025-02-15 11:16:35,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25517.77 MB 2025-02-15 11:16:35,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:16:35,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30828.13 MB 2025-02-15 11:16:35,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-15 11:16:35,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:16:35,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26225.56 MB 2025-02-15 11:16:35,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:16:35,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:16:35,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:16:35,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25930.66 MB 2025-02-15 11:16:35,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26158.65 MB 2025-02-15 11:16:35,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.99 MB 2025-02-15 11:16:35,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-15 11:16:35,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-15 11:16:35,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:16:35,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26348.98 MB 2025-02-15 11:16:35,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:16:35,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:16:35,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.62 seconds 2025-02-15 11:16:35,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14829.20 MB 2025-02-15 11:16:35,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26359.28 MB 2025-02-15 11:16:35,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11530.08 MB 2025-02-15 11:16:35,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38835.06 MB 2025-02-15 11:16:35,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-15 11:16:35,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7589.59 MB 2025-02-15 11:16:35,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26359.28 MB 2025-02-15 11:16:35,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:16:35,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:16:35,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:16:35,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26359.28 MB 2025-02-15 11:16:35,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19828.01 MB 2025-02-15 11:16:35,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6531.28 MB 2025-02-15 11:16:35,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-15 11:16:35,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-15 11:16:35,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:16:35,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28865.64 MB 2025-02-15 11:16:35,719 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 11:16:35,719 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:16:35,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:16:35,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:16:35,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:16:35,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:16:35,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19828.01 MB 2025-02-15 11:16:35,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.78 MB 2025-02-15 11:16:35,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 11:16:35,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-15 11:16:35,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41710.26 MB 2025-02-15 11:16:35,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 11:16:35,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28248.78 MB 2025-02-15 11:16:35,888 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 11:16:35,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:35,890 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:16:35,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:35,891 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:16:35,895 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:16:35,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:35,897 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:16:35,897 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:16:58,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:58,642 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:16:58,650 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:16:58,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:58,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1353, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:16:58,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:16:58,659 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1353, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:17:19,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:17:19,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:17:19,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.01 seconds 2025-02-15 11:17:19,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:19,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22396.63 MB 2025-02-15 11:17:19,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27184.82 MB 2025-02-15 11:17:19,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4788.19 MB 2025-02-15 11:17:19,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50082.09 MB 2025-02-15 11:17:19,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38012.98 MB 2025-02-15 11:17:19,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12069.11 MB 2025-02-15 11:17:19,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36171.35 MB 2025-02-15 11:17:19,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:17:19,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:17:19,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:17:19,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:19,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27184.82 MB 2025-02-15 11:17:19,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22811.68 MB 2025-02-15 11:17:19,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4373.14 MB 2025-02-15 11:17:19,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38012.98 MB 2025-02-15 11:17:19,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47143.98 MB 2025-02-15 11:17:19,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9131.00 MB 2025-02-15 11:17:19,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40917.23 MB 2025-02-15 11:17:21,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:17:21,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:17:21,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:17:21,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:21,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22811.68 MB 2025-02-15 11:17:21,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23342.52 MB 2025-02-15 11:17:21,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:17:21,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47143.98 MB 2025-02-15 11:17:21,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29037.17 MB 2025-02-15 11:17:21,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18106.81 MB 2025-02-15 11:17:21,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27321.06 MB 2025-02-15 11:17:21,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:17:21,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:17:21,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:17:21,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:21,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23342.52 MB 2025-02-15 11:17:21,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25232.05 MB 2025-02-15 11:17:21,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:17:21,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 11:17:21,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29980.88 MB 2025-02-15 11:17:21,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:17:21,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26649.48 MB 2025-02-15 11:17:21,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:17:21,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:17:21,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:17:21,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:21,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25232.05 MB 2025-02-15 11:17:21,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27473.91 MB 2025-02-15 11:17:21,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:17:21,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29980.88 MB 2025-02-15 11:17:21,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35643.20 MB 2025-02-15 11:17:21,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:17:21,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33018.19 MB 2025-02-15 11:17:21,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:17:21,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:17:21,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:17:21,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:21,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23342.52 MB 2025-02-15 11:17:21,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27473.91 MB 2025-02-15 11:17:21,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:17:21,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 11:17:21,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35643.20 MB 2025-02-15 11:17:21,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:17:21,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33018.19 MB 2025-02-15 11:17:22,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:17:22,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:17:22,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:17:22,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:22,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.45 MB 2025-02-15 11:17:22,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29774.45 MB 2025-02-15 11:17:22,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:17:22,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35643.20 MB 2025-02-15 11:17:22,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:17:22,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:17:22,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30482.24 MB 2025-02-15 11:17:22,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:17:22,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:17:22,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:17:22,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:22,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30187.34 MB 2025-02-15 11:17:22,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30417.25 MB 2025-02-15 11:17:22,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.91 MB 2025-02-15 11:17:22,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:17:22,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:17:22,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:22,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30624.15 MB 2025-02-15 11:17:22,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:17:22,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:17:22,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.44 seconds 2025-02-15 11:17:22,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:22,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17682.67 MB 2025-02-15 11:17:22,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30618.32 MB 2025-02-15 11:17:22,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12935.65 MB 2025-02-15 11:17:22,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50082.09 MB 2025-02-15 11:17:22,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:17:22,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14023.66 MB 2025-02-15 11:17:22,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30624.15 MB 2025-02-15 11:17:22,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:17:22,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:17:22,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:17:22,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:22,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30618.32 MB 2025-02-15 11:17:22,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22687.06 MB 2025-02-15 11:17:22,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7931.26 MB 2025-02-15 11:17:22,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:17:22,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36058.43 MB 2025-02-15 11:17:22,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:22,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33129.99 MB 2025-02-15 11:17:22,391 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:17:22,391 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:17:22,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:17:22,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:17:22,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:17:22,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:22,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22687.06 MB 2025-02-15 11:17:22,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31126.08 MB 2025-02-15 11:17:22,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:17:22,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36058.43 MB 2025-02-15 11:17:22,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44449.14 MB 2025-02-15 11:17:22,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:17:22,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31126.08 MB 2025-02-15 11:17:22,555 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:17:22,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:22,557 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:17:22,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:22,558 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:17:22,562 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:17:22,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:22,563 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:17:22,563 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:17:33,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:33,719 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:17:33,726 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:17:33,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:33,731 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:17:33,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:33,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:17:39,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:17:39,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:17:39,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.03 seconds 2025-02-15 11:17:39,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:39,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-15 11:17:39,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-15 11:17:39,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-15 11:17:39,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57034.15 MB 2025-02-15 11:17:39,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19312.67 MB 2025-02-15 11:17:39,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37721.47 MB 2025-02-15 11:17:39,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-15 11:17:39,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:17:39,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:17:39,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 11:17:39,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:39,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-15 11:17:39,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17390.11 MB 2025-02-15 11:17:39,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.21 MB 2025-02-15 11:17:39,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19312.67 MB 2025-02-15 11:17:39,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23601.35 MB 2025-02-15 11:17:39,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4288.68 MB 2025-02-15 11:17:39,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21853.23 MB 2025-02-15 11:17:41,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:17:41,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:17:41,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.70 seconds 2025-02-15 11:17:41,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17390.11 MB 2025-02-15 11:17:41,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17853.27 MB 2025-02-15 11:17:41,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 463.16 MB 2025-02-15 11:17:41,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23601.35 MB 2025-02-15 11:17:41,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20203.96 MB 2025-02-15 11:17:41,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3397.39 MB 2025-02-15 11:17:41,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21815.60 MB 2025-02-15 11:17:41,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:17:41,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:17:41,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:17:41,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17853.27 MB 2025-02-15 11:17:41,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19503.73 MB 2025-02-15 11:17:41,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1650.46 MB 2025-02-15 11:17:41,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20203.96 MB 2025-02-15 11:17:41,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22676.50 MB 2025-02-15 11:17:41,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2472.54 MB 2025-02-15 11:17:41,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20741.22 MB 2025-02-15 11:17:41,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:17:41,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:17:41,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:17:41,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19503.73 MB 2025-02-15 11:17:41,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21460.54 MB 2025-02-15 11:17:41,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1956.81 MB 2025-02-15 11:17:41,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22676.50 MB 2025-02-15 11:17:41,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28244.44 MB 2025-02-15 11:17:41,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5567.94 MB 2025-02-15 11:17:41,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26301.07 MB 2025-02-15 11:17:41,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:17:41,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:17:41,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 11:17:41,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17853.27 MB 2025-02-15 11:17:41,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21460.54 MB 2025-02-15 11:17:41,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3607.27 MB 2025-02-15 11:17:41,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20203.96 MB 2025-02-15 11:17:41,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28244.44 MB 2025-02-15 11:17:41,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8040.48 MB 2025-02-15 11:17:41,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26301.07 MB 2025-02-15 11:17:41,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:17:41,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:17:41,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 11:17:41,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22798.56 MB 2025-02-15 11:17:41,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23467.77 MB 2025-02-15 11:17:41,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 669.21 MB 2025-02-15 11:17:41,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28244.44 MB 2025-02-15 11:17:41,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28603.06 MB 2025-02-15 11:17:41,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-15 11:17:41,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24085.31 MB 2025-02-15 11:17:41,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:17:41,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:17:41,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:17:41,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23828.02 MB 2025-02-15 11:17:41,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24045.88 MB 2025-02-15 11:17:41,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.87 MB 2025-02-15 11:17:41,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28603.06 MB 2025-02-15 11:17:41,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28603.06 MB 2025-02-15 11:17:41,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:41,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24215.77 MB 2025-02-15 11:17:41,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:17:41,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:17:41,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.14 seconds 2025-02-15 11:17:41,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:41,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-15 11:17:41,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24246.96 MB 2025-02-15 11:17:41,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9957.79 MB 2025-02-15 11:17:41,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57034.15 MB 2025-02-15 11:17:41,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28603.06 MB 2025-02-15 11:17:41,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28431.09 MB 2025-02-15 11:17:41,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24246.96 MB 2025-02-15 11:17:42,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:17:42,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:17:42,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:17:42,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:42,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24246.96 MB 2025-02-15 11:17:42,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27260.99 MB 2025-02-15 11:17:42,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 11:17:42,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28603.06 MB 2025-02-15 11:17:42,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28603.06 MB 2025-02-15 11:17:42,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:42,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27562.36 MB 2025-02-15 11:17:42,161 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:17:42,161 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:17:42,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:17:42,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:17:42,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:17:42,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:42,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19052.55 MB 2025-02-15 11:17:42,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27491.57 MB 2025-02-15 11:17:42,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:17:42,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28603.06 MB 2025-02-15 11:17:42,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-15 11:17:42,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:17:42,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27491.57 MB 2025-02-15 11:17:42,326 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:17:42,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:42,327 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:17:42,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:42,328 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:17:42,333 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:17:42,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:42,334 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:17:42,334 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:17:52,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:52,207 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:17:52,212 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:17:52,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:52,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:17:52,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:52,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:17:55,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:17:55,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:17:55,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.87 seconds 2025-02-15 11:17:55,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-15 11:17:55,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-15 11:17:55,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-15 11:17:55,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51678.02 MB 2025-02-15 11:17:55,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32166.12 MB 2025-02-15 11:17:55,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.28 MB 2025-02-15 11:17:55,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:17:55,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:17:55,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:17:55,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-15 11:17:55,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14596.10 MB 2025-02-15 11:17:55,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -284.90 MB 2025-02-15 11:17:55,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16243.53 MB 2025-02-15 11:17:55,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:17:55,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:17:55,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.48 seconds 2025-02-15 11:17:55,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14596.10 MB 2025-02-15 11:17:55,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14724.83 MB 2025-02-15 11:17:55,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 128.73 MB 2025-02-15 11:17:55,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18681.85 MB 2025-02-15 11:17:55,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:17:55,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:17:55,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:17:55,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14724.76 MB 2025-02-15 11:17:55,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15182.87 MB 2025-02-15 11:17:55,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.10 MB 2025-02-15 11:17:55,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15526.60 MB 2025-02-15 11:17:55,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:17:55,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:17:55,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:17:55,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15182.87 MB 2025-02-15 11:17:55,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.28 MB 2025-02-15 11:17:55,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 556.41 MB 2025-02-15 11:17:55,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17071.00 MB 2025-02-15 11:17:55,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:17:55,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:17:55,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:17:55,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14724.76 MB 2025-02-15 11:17:55,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.28 MB 2025-02-15 11:17:55,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.51 MB 2025-02-15 11:17:55,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19511.90 MB 2025-02-15 11:17:55,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17071.00 MB 2025-02-15 11:17:55,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:17:55,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:17:55,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 11:17:55,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16276.44 MB 2025-02-15 11:17:55,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16510.12 MB 2025-02-15 11:17:55,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.68 MB 2025-02-15 11:17:55,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19511.90 MB 2025-02-15 11:17:55,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-15 11:17:55,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-15 11:17:55,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16681.76 MB 2025-02-15 11:17:55,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:17:55,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:17:55,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:17:55,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16657.93 MB 2025-02-15 11:17:55,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16883.26 MB 2025-02-15 11:17:55,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.33 MB 2025-02-15 11:17:55,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-15 11:17:55,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-15 11:17:55,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:55,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16883.26 MB 2025-02-15 11:17:55,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:17:55,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:17:55,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.52 seconds 2025-02-15 11:17:55,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:55,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-15 11:17:55,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17084.06 MB 2025-02-15 11:17:55,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3481.26 MB 2025-02-15 11:17:55,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51678.02 MB 2025-02-15 11:17:55,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-15 11:17:55,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32019.32 MB 2025-02-15 11:17:55,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17084.06 MB 2025-02-15 11:17:56,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:17:56,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:17:56,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:17:56,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:56,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14251.53 MB 2025-02-15 11:17:56,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17261.51 MB 2025-02-15 11:17:56,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.98 MB 2025-02-15 11:17:56,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-15 11:17:56,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-15 11:17:56,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:17:56,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17562.47 MB 2025-02-15 11:17:56,030 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 11:17:56,030 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:17:56,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:17:56,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:17:56,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:17:56,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:17:56,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17261.51 MB 2025-02-15 11:17:56,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25688.84 MB 2025-02-15 11:17:56,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 11:17:56,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-15 11:17:56,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30133.98 MB 2025-02-15 11:17:56,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 11:17:56,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25688.84 MB 2025-02-15 11:17:56,194 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 11:17:56,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:56,196 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:17:56,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:56,197 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:17:56,201 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:17:56,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:17:56,202 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:17:56,203 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:19:16,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:16,965 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:19:16,970 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:19:16,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:16,974 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 176, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:19:16,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:16,975 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 176, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:19:19,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:19:19,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:19:19,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.70 seconds 2025-02-15 11:19:19,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:19,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-15 11:19:19,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14817.96 MB 2025-02-15 11:19:19,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 622.85 MB 2025-02-15 11:19:19,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38514.20 MB 2025-02-15 11:19:19,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19356.71 MB 2025-02-15 11:19:19,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19157.48 MB 2025-02-15 11:19:19,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23666.47 MB 2025-02-15 11:19:19,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:19:19,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:19:19,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:19:19,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:19,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14817.96 MB 2025-02-15 11:19:19,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15119.73 MB 2025-02-15 11:19:19,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 301.77 MB 2025-02-15 11:19:19,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19356.71 MB 2025-02-15 11:19:19,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19356.71 MB 2025-02-15 11:19:19,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:19:19,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17290.13 MB 2025-02-15 11:19:20,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:19:20,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:19:20,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 11:19:20,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15119.73 MB 2025-02-15 11:19:20,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15353.30 MB 2025-02-15 11:19:20,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-15 11:19:20,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19356.71 MB 2025-02-15 11:19:20,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18943.57 MB 2025-02-15 11:19:20,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -413.14 MB 2025-02-15 11:19:20,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19290.42 MB 2025-02-15 11:19:20,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:19:20,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:19:20,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:19:20,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-15 11:19:20,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16184.43 MB 2025-02-15 11:19:20,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-15 11:19:20,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18943.57 MB 2025-02-15 11:19:20,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18943.57 MB 2025-02-15 11:19:20,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:19:20,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16808.10 MB 2025-02-15 11:19:20,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:19:20,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:19:20,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:19:20,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16184.43 MB 2025-02-15 11:19:20,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-15 11:19:20,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-15 11:19:20,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18943.57 MB 2025-02-15 11:19:20,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21019.75 MB 2025-02-15 11:19:20,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 11:19:20,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-15 11:19:20,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:19:20,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:19:20,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:19:20,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-15 11:19:20,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-15 11:19:20,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-15 11:19:20,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18943.57 MB 2025-02-15 11:19:20,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21019.75 MB 2025-02-15 11:19:20,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2076.18 MB 2025-02-15 11:19:20,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-15 11:19:20,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:19:20,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:19:20,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:19:20,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17845.64 MB 2025-02-15 11:19:20,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18183.12 MB 2025-02-15 11:19:20,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-15 11:19:20,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21019.75 MB 2025-02-15 11:19:20,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21202.21 MB 2025-02-15 11:19:20,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 11:19:20,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18501.02 MB 2025-02-15 11:19:20,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:19:20,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:19:20,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:19:20,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18364.80 MB 2025-02-15 11:19:20,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18594.05 MB 2025-02-15 11:19:20,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.25 MB 2025-02-15 11:19:20,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21202.21 MB 2025-02-15 11:19:20,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21202.21 MB 2025-02-15 11:19:20,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:19:20,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18632.04 MB 2025-02-15 11:19:20,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:19:20,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:19:20,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.76 seconds 2025-02-15 11:19:20,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:20,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13581.90 MB 2025-02-15 11:19:20,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18795.10 MB 2025-02-15 11:19:20,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5213.19 MB 2025-02-15 11:19:20,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38514.20 MB 2025-02-15 11:19:20,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21202.21 MB 2025-02-15 11:19:20,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17311.99 MB 2025-02-15 11:19:20,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18795.10 MB 2025-02-15 11:19:20,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:19:21,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:19:21,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:19:21,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:21,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18795.10 MB 2025-02-15 11:19:21,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17529.81 MB 2025-02-15 11:19:21,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1265.29 MB 2025-02-15 11:19:21,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21202.21 MB 2025-02-15 11:19:21,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21202.21 MB 2025-02-15 11:19:21,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:19:21,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19031.17 MB 2025-02-15 11:19:21,018 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 11:19:21,018 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 11:19:21,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:19:21,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:19:21,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:19:21,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:19:21,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17529.81 MB 2025-02-15 11:19:21,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25968.65 MB 2025-02-15 11:19:21,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 11:19:21,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21202.21 MB 2025-02-15 11:19:21,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31687.97 MB 2025-02-15 11:19:21,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 11:19:21,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25968.65 MB 2025-02-15 11:19:21,184 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 11:19:21,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:21,185 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:19:21,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:21,186 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:19:21,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:19:21,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:19:21,192 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:19:21,192 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 11:20:22,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:22,187 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:20:22,193 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:20:22,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:22,196 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1931, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:20:22,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:22,197 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1931, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:20:51,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:20:51,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:20:51,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.73 seconds 2025-02-15 11:20:51,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:51,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26424.22 MB 2025-02-15 11:20:51,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33258.84 MB 2025-02-15 11:20:51,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6834.62 MB 2025-02-15 11:20:51,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40076.57 MB 2025-02-15 11:20:51,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40091.25 MB 2025-02-15 11:20:51,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14.68 MB 2025-02-15 11:20:51,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42237.38 MB 2025-02-15 11:20:52,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:20:52,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:20:52,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 11:20:52,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:52,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33258.84 MB 2025-02-15 11:20:52,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25816.52 MB 2025-02-15 11:20:52,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7442.32 MB 2025-02-15 11:20:52,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40091.25 MB 2025-02-15 11:20:52,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54475.62 MB 2025-02-15 11:20:52,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14384.37 MB 2025-02-15 11:20:52,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52758.26 MB 2025-02-15 11:20:54,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:20:54,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:20:54,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:20:54,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25816.52 MB 2025-02-15 11:20:54,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26347.36 MB 2025-02-15 11:20:54,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:20:54,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54475.62 MB 2025-02-15 11:20:54,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:20:54,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19803.41 MB 2025-02-15 11:20:54,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30325.91 MB 2025-02-15 11:20:54,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:20:54,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:20:54,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:20:54,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-15 11:20:54,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28236.89 MB 2025-02-15 11:20:54,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:20:54,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:20:54,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:20:54,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:20:54,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29654.32 MB 2025-02-15 11:20:54,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:20:54,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:20:54,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:20:54,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28236.89 MB 2025-02-15 11:20:54,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-15 11:20:54,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:20:54,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:20:54,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39390.81 MB 2025-02-15 11:20:54,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 11:20:54,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-15 11:20:54,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:20:54,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:20:54,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:20:54,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-15 11:20:54,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-15 11:20:54,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:20:54,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:20:54,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39390.81 MB 2025-02-15 11:20:54,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 11:20:54,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-15 11:20:54,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:20:54,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:20:54,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:20:54,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32012.29 MB 2025-02-15 11:20:54,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32779.29 MB 2025-02-15 11:20:54,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:20:54,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39390.81 MB 2025-02-15 11:20:54,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-15 11:20:54,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:20:54,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.08 MB 2025-02-15 11:20:54,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:20:54,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:20:54,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:20:54,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33192.18 MB 2025-02-15 11:20:54,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33420.16 MB 2025-02-15 11:20:54,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-15 11:20:54,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-15 11:20:54,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-15 11:20:54,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:20:54,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33645.95 MB 2025-02-15 11:20:54,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:20:54,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:20:54,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.28 seconds 2025-02-15 11:20:54,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19696.46 MB 2025-02-15 11:20:54,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33620.05 MB 2025-02-15 11:20:54,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13923.59 MB 2025-02-15 11:20:54,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40076.57 MB 2025-02-15 11:20:54,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-15 11:20:54,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -270.53 MB 2025-02-15 11:20:54,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33645.95 MB 2025-02-15 11:20:54,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:20:54,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:20:54,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:20:54,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33620.05 MB 2025-02-15 11:20:54,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24683.53 MB 2025-02-15 11:20:54,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8936.52 MB 2025-02-15 11:20:54,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-15 11:20:54,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-15 11:20:54,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:20:54,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36116.97 MB 2025-02-15 11:20:54,764 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 11:20:54,764 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:20:54,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:20:54,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:20:54,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:20:54,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:20:54,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24683.53 MB 2025-02-15 11:20:54,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33072.67 MB 2025-02-15 11:20:54,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 11:20:54,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-15 11:20:54,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48148.51 MB 2025-02-15 11:20:54,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 11:20:54,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33072.67 MB 2025-02-15 11:20:54,931 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 11:20:54,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:54,932 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:20:54,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:54,933 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:20:54,938 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:20:54,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:20:54,939 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:20:54,939 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:22:13,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:13,082 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:22:13,087 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:22:13,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:13,091 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1838, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:22:13,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:13,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1838, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:22:41,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:22:41,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:22:41,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.39 seconds 2025-02-15 11:22:41,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:41,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25776.18 MB 2025-02-15 11:22:41,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32281.55 MB 2025-02-15 11:22:41,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6505.37 MB 2025-02-15 11:22:41,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56490.98 MB 2025-02-15 11:22:41,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39692.80 MB 2025-02-15 11:22:41,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16798.19 MB 2025-02-15 11:22:41,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41136.36 MB 2025-02-15 11:22:41,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:22:41,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:22:41,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:22:41,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:41,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32281.55 MB 2025-02-15 11:22:41,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25333.04 MB 2025-02-15 11:22:41,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6948.51 MB 2025-02-15 11:22:41,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39692.80 MB 2025-02-15 11:22:41,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51820.63 MB 2025-02-15 11:22:41,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12127.83 MB 2025-02-15 11:22:41,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48393.49 MB 2025-02-15 11:22:43,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:22:43,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:22:43,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:22:43,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.04 MB 2025-02-15 11:22:43,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25863.88 MB 2025-02-15 11:22:43,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:22:43,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51820.63 MB 2025-02-15 11:22:43,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30431.77 MB 2025-02-15 11:22:43,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21388.85 MB 2025-02-15 11:22:43,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29843.47 MB 2025-02-15 11:22:43,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:22:43,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:22:43,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:22:43,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-15 11:22:43,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27753.41 MB 2025-02-15 11:22:43,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:22:43,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30431.77 MB 2025-02-15 11:22:43,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30431.77 MB 2025-02-15 11:22:43,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:22:43,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29170.84 MB 2025-02-15 11:22:43,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:22:43,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:22:43,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:22:43,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27753.41 MB 2025-02-15 11:22:43,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-15 11:22:43,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:22:43,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30431.77 MB 2025-02-15 11:22:43,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37746.64 MB 2025-02-15 11:22:43,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:22:43,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-15 11:22:43,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:22:43,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:22:43,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:22:43,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-15 11:22:43,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-15 11:22:43,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:22:43,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30431.77 MB 2025-02-15 11:22:43,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37746.64 MB 2025-02-15 11:22:43,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:22:43,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-15 11:22:43,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:22:43,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:22:43,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:22:43,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31528.81 MB 2025-02-15 11:22:43,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32295.81 MB 2025-02-15 11:22:43,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:22:43,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37746.64 MB 2025-02-15 11:22:43,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 11:22:43,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:22:43,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33003.60 MB 2025-02-15 11:22:43,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:22:43,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:22:43,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:22:43,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32708.70 MB 2025-02-15 11:22:43,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32936.96 MB 2025-02-15 11:22:43,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-15 11:22:43,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 11:22:43,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 11:22:43,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:22:43,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33178.79 MB 2025-02-15 11:22:43,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:22:43,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:22:43,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.89 seconds 2025-02-15 11:22:43,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:43,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19372.45 MB 2025-02-15 11:22:43,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33137.82 MB 2025-02-15 11:22:43,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13765.37 MB 2025-02-15 11:22:43,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56490.98 MB 2025-02-15 11:22:43,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 11:22:43,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18329.11 MB 2025-02-15 11:22:43,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33178.79 MB 2025-02-15 11:22:44,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:22:44,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:22:44,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:22:44,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:44,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33137.82 MB 2025-02-15 11:22:44,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24373.41 MB 2025-02-15 11:22:44,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8764.41 MB 2025-02-15 11:22:44,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 11:22:44,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-15 11:22:44,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:22:44,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35646.72 MB 2025-02-15 11:22:44,272 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 11:22:44,272 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:22:44,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:22:44,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:22:44,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:22:44,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:22:44,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24373.41 MB 2025-02-15 11:22:44,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32803.81 MB 2025-02-15 11:22:44,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-15 11:22:44,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-15 11:22:44,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46542.09 MB 2025-02-15 11:22:44,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 11:22:44,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32803.81 MB 2025-02-15 11:22:44,440 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 11:22:44,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:44,442 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:22:44,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:44,443 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:22:44,447 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:22:44,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:44,448 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:22:44,448 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:22:53,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:53,069 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:22:53,074 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:22:53,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:53,077 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1743, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:22:53,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:22:53,078 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1743, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:23:20,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:23:20,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:23:20,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.31 seconds 2025-02-15 11:23:20,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:20,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25114.21 MB 2025-02-15 11:23:20,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31282.59 MB 2025-02-15 11:23:20,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6168.38 MB 2025-02-15 11:23:20,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54922.31 MB 2025-02-15 11:23:20,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39369.83 MB 2025-02-15 11:23:20,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15552.48 MB 2025-02-15 11:23:20,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40247.89 MB 2025-02-15 11:23:20,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:23:20,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:23:20,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:23:20,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:20,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31282.59 MB 2025-02-15 11:23:20,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24839.16 MB 2025-02-15 11:23:20,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6443.42 MB 2025-02-15 11:23:20,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39369.83 MB 2025-02-15 11:23:20,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52453.97 MB 2025-02-15 11:23:20,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13084.13 MB 2025-02-15 11:23:20,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48827.21 MB 2025-02-15 11:23:22,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:23:22,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:23:22,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:23:22,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24839.16 MB 2025-02-15 11:23:22,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25370.01 MB 2025-02-15 11:23:22,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:23:22,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52453.97 MB 2025-02-15 11:23:22,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30444.36 MB 2025-02-15 11:23:22,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22009.61 MB 2025-02-15 11:23:22,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29348.55 MB 2025-02-15 11:23:22,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:23:22,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:23:22,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:23:22,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-15 11:23:22,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27259.54 MB 2025-02-15 11:23:22,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:23:22,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30444.36 MB 2025-02-15 11:23:22,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30444.36 MB 2025-02-15 11:23:22,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:22,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.97 MB 2025-02-15 11:23:22,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:23:22,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:23:22,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:23:22,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27259.54 MB 2025-02-15 11:23:22,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-15 11:23:22,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:23:22,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30444.36 MB 2025-02-15 11:23:22,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37050.38 MB 2025-02-15 11:23:22,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:23:22,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-15 11:23:22,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:23:22,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:23:22,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:23:22,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-15 11:23:22,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-15 11:23:22,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:23:22,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30444.36 MB 2025-02-15 11:23:22,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37050.38 MB 2025-02-15 11:23:22,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:23:22,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-15 11:23:22,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:23:22,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:23:22,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:23:22,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31034.94 MB 2025-02-15 11:23:22,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31801.94 MB 2025-02-15 11:23:22,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:23:22,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37050.38 MB 2025-02-15 11:23:22,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37467.72 MB 2025-02-15 11:23:22,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:23:22,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32509.73 MB 2025-02-15 11:23:22,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:23:22,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:23:22,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:23:22,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32214.83 MB 2025-02-15 11:23:22,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32443.06 MB 2025-02-15 11:23:22,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.24 MB 2025-02-15 11:23:22,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37467.72 MB 2025-02-15 11:23:22,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37467.72 MB 2025-02-15 11:23:22,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:22,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32666.98 MB 2025-02-15 11:23:22,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:23:22,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:23:22,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.82 seconds 2025-02-15 11:23:22,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:22,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19041.46 MB 2025-02-15 11:23:22,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32643.92 MB 2025-02-15 11:23:22,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13602.46 MB 2025-02-15 11:23:22,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54922.31 MB 2025-02-15 11:23:22,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37467.72 MB 2025-02-15 11:23:22,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17454.60 MB 2025-02-15 11:23:22,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32666.98 MB 2025-02-15 11:23:23,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:23:23,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:23:23,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:23:23,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:23,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32643.92 MB 2025-02-15 11:23:23,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24031.37 MB 2025-02-15 11:23:23,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8612.54 MB 2025-02-15 11:23:23,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37467.72 MB 2025-02-15 11:23:23,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37467.72 MB 2025-02-15 11:23:23,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:23,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35143.29 MB 2025-02-15 11:23:23,195 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 11:23:23,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:23:23,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:23:23,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:23:23,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:23:23,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:23,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24031.37 MB 2025-02-15 11:23:23,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32428.77 MB 2025-02-15 11:23:23,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-15 11:23:23,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37467.72 MB 2025-02-15 11:23:23,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41643.15 MB 2025-02-15 11:23:23,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 11:23:23,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32428.77 MB 2025-02-15 11:23:23,360 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 11:23:23,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:23,362 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:23:23,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:23,363 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:23:23,367 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:23:23,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:23,369 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:23:23,369 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:23:32,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:32,908 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:23:32,915 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:23:32,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:32,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:23:32,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:32,923 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:23:35,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:23:35,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:23:35,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.62 seconds 2025-02-15 11:23:35,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:35,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-15 11:23:35,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-15 11:23:35,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 11:23:35,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49994.01 MB 2025-02-15 11:23:35,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18362.66 MB 2025-02-15 11:23:35,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31631.34 MB 2025-02-15 11:23:35,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-15 11:23:35,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:23:35,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:23:35,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:23:35,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:35,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-15 11:23:35,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-15 11:23:35,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-15 11:23:35,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18362.66 MB 2025-02-15 11:23:35,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18362.66 MB 2025-02-15 11:23:35,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:35,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16921.83 MB 2025-02-15 11:23:36,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:23:36,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:23:36,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-15 11:23:36,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-15 11:23:36,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-15 11:23:36,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 11:23:36,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18362.66 MB 2025-02-15 11:23:36,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18293.46 MB 2025-02-15 11:23:36,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -69.21 MB 2025-02-15 11:23:36,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.09 MB 2025-02-15 11:23:36,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:23:36,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:23:36,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:23:36,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 11:23:36,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-15 11:23:36,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 11:23:36,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18293.46 MB 2025-02-15 11:23:36,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18295.55 MB 2025-02-15 11:23:36,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 11:23:36,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-15 11:23:36,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:23:36,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:23:36,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:23:36,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-15 11:23:36,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 11:23:36,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 11:23:36,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18295.55 MB 2025-02-15 11:23:36,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20394.80 MB 2025-02-15 11:23:36,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2099.25 MB 2025-02-15 11:23:36,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19045.19 MB 2025-02-15 11:23:36,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:23:36,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:23:36,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:23:36,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-15 11:23:36,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-15 11:23:36,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 11:23:36,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18293.46 MB 2025-02-15 11:23:36,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20394.80 MB 2025-02-15 11:23:36,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2101.35 MB 2025-02-15 11:23:36,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19045.19 MB 2025-02-15 11:23:36,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:23:36,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:23:36,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:23:36,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-15 11:23:36,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.63 MB 2025-02-15 11:23:36,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-15 11:23:36,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20394.80 MB 2025-02-15 11:23:36,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20562.58 MB 2025-02-15 11:23:36,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 11:23:36,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18032.37 MB 2025-02-15 11:23:36,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:23:36,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:23:36,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:23:36,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17905.82 MB 2025-02-15 11:23:36,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18132.70 MB 2025-02-15 11:23:36,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.88 MB 2025-02-15 11:23:36,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20562.58 MB 2025-02-15 11:23:36,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20562.58 MB 2025-02-15 11:23:36,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:36,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18143.61 MB 2025-02-15 11:23:36,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:23:36,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:23:36,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.74 seconds 2025-02-15 11:23:36,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-15 11:23:36,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18333.38 MB 2025-02-15 11:23:36,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4803.74 MB 2025-02-15 11:23:36,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49994.01 MB 2025-02-15 11:23:36,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20562.58 MB 2025-02-15 11:23:36,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29431.43 MB 2025-02-15 11:23:36,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18333.38 MB 2025-02-15 11:23:36,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:23:36,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:23:36,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 11:23:36,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18333.38 MB 2025-02-15 11:23:36,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17400.94 MB 2025-02-15 11:23:36,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -932.43 MB 2025-02-15 11:23:36,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20562.58 MB 2025-02-15 11:23:36,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20562.58 MB 2025-02-15 11:23:36,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:36,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19135.54 MB 2025-02-15 11:23:36,985 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 11:23:36,986 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:23:36,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:23:36,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:23:36,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 11:23:36,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:36,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17400.94 MB 2025-02-15 11:23:36,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25823.27 MB 2025-02-15 11:23:36,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 11:23:36,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20562.58 MB 2025-02-15 11:23:36,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31031.56 MB 2025-02-15 11:23:36,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 11:23:36,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25823.27 MB 2025-02-15 11:23:37,246 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 11:23:37,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:37,250 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:23:37,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:37,252 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:23:37,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:23:37,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:37,261 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:23:37,262 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:23:49,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:49,418 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:23:49,425 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:23:49,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:49,432 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:23:49,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:49,433 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:23:52,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:23:52,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:23:52,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.98 seconds 2025-02-15 11:23:52,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:52,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-15 11:23:52,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-15 11:23:52,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-15 11:23:52,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43591.40 MB 2025-02-15 11:23:52,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19488.83 MB 2025-02-15 11:23:52,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24102.57 MB 2025-02-15 11:23:52,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-15 11:23:52,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:23:52,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:23:52,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:23:52,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:52,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-15 11:23:52,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15241.88 MB 2025-02-15 11:23:52,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.85 MB 2025-02-15 11:23:52,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19488.83 MB 2025-02-15 11:23:52,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19488.83 MB 2025-02-15 11:23:52,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:52,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17535.59 MB 2025-02-15 11:23:53,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:23:53,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:23:53,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 11:23:53,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.88 MB 2025-02-15 11:23:53,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15488.72 MB 2025-02-15 11:23:53,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-15 11:23:53,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19488.83 MB 2025-02-15 11:23:53,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17987.27 MB 2025-02-15 11:23:53,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1501.56 MB 2025-02-15 11:23:53,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19413.61 MB 2025-02-15 11:23:53,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:23:53,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:23:53,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:23:53,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-15 11:23:53,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.14 MB 2025-02-15 11:23:53,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-15 11:23:53,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17987.27 MB 2025-02-15 11:23:53,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18427.67 MB 2025-02-15 11:23:53,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 440.40 MB 2025-02-15 11:23:53,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.25 MB 2025-02-15 11:23:53,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:23:53,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:23:53,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:23:53,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.14 MB 2025-02-15 11:23:53,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-15 11:23:53,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-15 11:23:53,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18427.67 MB 2025-02-15 11:23:53,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21290.29 MB 2025-02-15 11:23:53,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2862.61 MB 2025-02-15 11:23:53,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19988.48 MB 2025-02-15 11:23:53,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:23:53,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:23:53,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 11:23:53,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-15 11:23:53,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-15 11:23:53,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-15 11:23:53,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17987.27 MB 2025-02-15 11:23:53,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21290.29 MB 2025-02-15 11:23:53,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 11:23:53,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19988.48 MB 2025-02-15 11:23:53,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:23:53,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:23:53,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:23:53,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18122.74 MB 2025-02-15 11:23:53,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18480.18 MB 2025-02-15 11:23:53,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 357.44 MB 2025-02-15 11:23:53,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21290.29 MB 2025-02-15 11:23:53,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21483.23 MB 2025-02-15 11:23:53,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 192.94 MB 2025-02-15 11:23:53,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18816.25 MB 2025-02-15 11:23:53,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:23:53,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:23:53,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:23:53,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18672.18 MB 2025-02-15 11:23:53,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18876.12 MB 2025-02-15 11:23:53,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.95 MB 2025-02-15 11:23:53,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21483.23 MB 2025-02-15 11:23:53,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21487.42 MB 2025-02-15 11:23:53,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 11:23:53,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18901.03 MB 2025-02-15 11:23:53,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:23:53,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:23:53,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.27 seconds 2025-02-15 11:23:53,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:53,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-15 11:23:53,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19077.20 MB 2025-02-15 11:23:53,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5460.45 MB 2025-02-15 11:23:53,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43591.40 MB 2025-02-15 11:23:53,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21487.42 MB 2025-02-15 11:23:53,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22103.98 MB 2025-02-15 11:23:53,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.20 MB 2025-02-15 11:23:54,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:23:54,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:23:54,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 11:23:54,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:54,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.20 MB 2025-02-15 11:23:54,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17611.47 MB 2025-02-15 11:23:54,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1465.73 MB 2025-02-15 11:23:54,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21487.42 MB 2025-02-15 11:23:54,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21487.42 MB 2025-02-15 11:23:54,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:23:54,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.21 MB 2025-02-15 11:23:54,022 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:23:54,022 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:23:54,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:23:54,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:23:54,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:23:54,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:23:54,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17611.47 MB 2025-02-15 11:23:54,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26050.49 MB 2025-02-15 11:23:54,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:23:54,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21487.42 MB 2025-02-15 11:23:54,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31977.37 MB 2025-02-15 11:23:54,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:23:54,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26050.49 MB 2025-02-15 11:23:54,281 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:23:54,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:54,284 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:23:54,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:54,286 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:23:54,293 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:23:54,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:23:54,296 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:23:54,296 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:25:54,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:54,019 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:25:54,026 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:25:54,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:54,034 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:25:54,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:54,035 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:25:57,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:25:57,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:25:57,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.63 seconds 2025-02-15 11:25:57,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:57,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-15 11:25:57,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15395.85 MB 2025-02-15 11:25:57,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.50 MB 2025-02-15 11:25:57,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44562.38 MB 2025-02-15 11:25:57,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19876.81 MB 2025-02-15 11:25:57,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24685.58 MB 2025-02-15 11:25:57,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24276.21 MB 2025-02-15 11:25:57,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:25:57,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:25:57,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:25:57,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:57,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15395.85 MB 2025-02-15 11:25:57,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15750.37 MB 2025-02-15 11:25:57,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.53 MB 2025-02-15 11:25:57,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19876.81 MB 2025-02-15 11:25:57,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20673.72 MB 2025-02-15 11:25:57,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 796.92 MB 2025-02-15 11:25:57,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18556.86 MB 2025-02-15 11:25:58,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:25:58,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:25:58,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.09 seconds 2025-02-15 11:25:58,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:58,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15750.37 MB 2025-02-15 11:25:58,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16048.97 MB 2025-02-15 11:25:58,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.60 MB 2025-02-15 11:25:58,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20673.72 MB 2025-02-15 11:25:58,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20233.32 MB 2025-02-15 11:25:58,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -440.40 MB 2025-02-15 11:25:58,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20006.00 MB 2025-02-15 11:25:58,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:25:58,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:25:58,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:25:58,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:58,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.97 MB 2025-02-15 11:25:58,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17111.58 MB 2025-02-15 11:25:58,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1062.60 MB 2025-02-15 11:25:58,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20233.32 MB 2025-02-15 11:25:58,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20233.32 MB 2025-02-15 11:25:58,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:25:58,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17908.88 MB 2025-02-15 11:25:58,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:25:58,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:25:58,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:25:58,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:58,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17111.58 MB 2025-02-15 11:25:58,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18372.65 MB 2025-02-15 11:25:58,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.07 MB 2025-02-15 11:25:58,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20233.32 MB 2025-02-15 11:25:58,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23429.38 MB 2025-02-15 11:25:58,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3196.06 MB 2025-02-15 11:25:58,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21493.11 MB 2025-02-15 11:25:58,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:25:58,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:25:58,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:25:58,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:58,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.97 MB 2025-02-15 11:25:58,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18372.65 MB 2025-02-15 11:25:58,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2323.68 MB 2025-02-15 11:25:58,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20233.32 MB 2025-02-15 11:25:58,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23429.38 MB 2025-02-15 11:25:58,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3196.06 MB 2025-02-15 11:25:58,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21493.11 MB 2025-02-15 11:25:59,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:25:59,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:25:59,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:25:59,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:59,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19235.27 MB 2025-02-15 11:25:59,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.54 MB 2025-02-15 11:25:59,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 433.27 MB 2025-02-15 11:25:59,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23429.38 MB 2025-02-15 11:25:59,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23662.17 MB 2025-02-15 11:25:59,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 232.78 MB 2025-02-15 11:25:59,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20067.03 MB 2025-02-15 11:25:59,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:25:59,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:25:59,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:25:59,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:59,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19900.80 MB 2025-02-15 11:25:59,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20128.92 MB 2025-02-15 11:25:59,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.12 MB 2025-02-15 11:25:59,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23662.17 MB 2025-02-15 11:25:59,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23664.26 MB 2025-02-15 11:25:59,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 11:25:59,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20205.64 MB 2025-02-15 11:25:59,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:25:59,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:25:59,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.99 seconds 2025-02-15 11:25:59,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:59,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13773.53 MB 2025-02-15 11:25:59,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20329.99 MB 2025-02-15 11:25:59,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6556.46 MB 2025-02-15 11:25:59,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44562.38 MB 2025-02-15 11:25:59,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23664.26 MB 2025-02-15 11:25:59,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20898.12 MB 2025-02-15 11:25:59,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20329.99 MB 2025-02-15 11:25:59,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:25:59,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:25:59,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 11:25:59,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:59,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14939.39 MB 2025-02-15 11:25:59,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17953.43 MB 2025-02-15 11:25:59,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 11:25:59,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23664.26 MB 2025-02-15 11:25:59,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23664.26 MB 2025-02-15 11:25:59,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:25:59,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18254.79 MB 2025-02-15 11:25:59,308 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:25:59,308 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:25:59,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:25:59,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:25:59,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:25:59,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:25:59,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17953.43 MB 2025-02-15 11:25:59,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26392.45 MB 2025-02-15 11:25:59,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:25:59,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23664.26 MB 2025-02-15 11:25:59,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34154.22 MB 2025-02-15 11:25:59,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:25:59,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26392.45 MB 2025-02-15 11:25:59,474 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:25:59,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:59,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:25:59,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:59,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:25:59,481 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:25:59,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:25:59,482 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:25:59,483 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:26:10,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:10,675 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:26:10,683 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:26:10,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:10,690 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2626, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:26:10,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:10,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2626, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:26:51,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:26:51,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:26:51,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.86 seconds 2025-02-15 11:26:51,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:51,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31268.45 MB 2025-02-15 11:26:51,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40561.72 MB 2025-02-15 11:26:51,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9293.27 MB 2025-02-15 11:26:51,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65038.97 MB 2025-02-15 11:26:51,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44071.65 MB 2025-02-15 11:26:51,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20967.33 MB 2025-02-15 11:26:51,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49854.99 MB 2025-02-15 11:26:51,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:26:51,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:26:51,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 11:26:51,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:51,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40561.72 MB 2025-02-15 11:26:51,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29431.36 MB 2025-02-15 11:26:51,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11130.36 MB 2025-02-15 11:26:51,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44071.65 MB 2025-02-15 11:26:51,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63594.04 MB 2025-02-15 11:26:51,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19522.39 MB 2025-02-15 11:26:51,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66725.88 MB 2025-02-15 11:26:53,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:26:53,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:26:53,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:26:53,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:53,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29431.36 MB 2025-02-15 11:26:53,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29962.20 MB 2025-02-15 11:26:53,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:26:53,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63594.04 MB 2025-02-15 11:26:53,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32254.20 MB 2025-02-15 11:26:53,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31339.84 MB 2025-02-15 11:26:53,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33940.75 MB 2025-02-15 11:26:53,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:26:53,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:26:53,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:26:53,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:53,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29962.20 MB 2025-02-15 11:26:53,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31851.73 MB 2025-02-15 11:26:53,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:26:53,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32254.20 MB 2025-02-15 11:26:53,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-15 11:26:53,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:26:53,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33269.16 MB 2025-02-15 11:26:54,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:26:54,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:26:54,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:26:54,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31851.73 MB 2025-02-15 11:26:54,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34093.59 MB 2025-02-15 11:26:54,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:26:54,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-15 11:26:54,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41219.52 MB 2025-02-15 11:26:54,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:26:54,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39637.87 MB 2025-02-15 11:26:54,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:26:54,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:26:54,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:26:54,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29962.20 MB 2025-02-15 11:26:54,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34093.59 MB 2025-02-15 11:26:54,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:26:54,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32254.20 MB 2025-02-15 11:26:54,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41219.52 MB 2025-02-15 11:26:54,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 11:26:54,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39637.87 MB 2025-02-15 11:26:54,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:26:54,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:26:54,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:26:54,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35627.13 MB 2025-02-15 11:26:54,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36394.13 MB 2025-02-15 11:26:54,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:26:54,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41219.52 MB 2025-02-15 11:26:54,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41636.86 MB 2025-02-15 11:26:54,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:26:54,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37101.92 MB 2025-02-15 11:26:54,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:26:54,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:26:54,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:26:54,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36807.02 MB 2025-02-15 11:26:54,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37035.69 MB 2025-02-15 11:26:54,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-15 11:26:54,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41636.86 MB 2025-02-15 11:26:54,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41636.86 MB 2025-02-15 11:26:54,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:26:54,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37236.61 MB 2025-02-15 11:26:54,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:26:54,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:26:54,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.51 seconds 2025-02-15 11:26:54,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22118.58 MB 2025-02-15 11:26:54,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37236.27 MB 2025-02-15 11:26:54,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15117.69 MB 2025-02-15 11:26:54,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55889.10 MB 2025-02-15 11:26:54,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41636.86 MB 2025-02-15 11:26:54,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14252.24 MB 2025-02-15 11:26:54,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37236.61 MB 2025-02-15 11:26:54,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:26:54,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:26:54,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:26:54,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37236.27 MB 2025-02-15 11:26:54,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27115.62 MB 2025-02-15 11:26:54,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10120.65 MB 2025-02-15 11:26:54,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41636.86 MB 2025-02-15 11:26:54,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41636.86 MB 2025-02-15 11:26:54,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:26:54,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39741.79 MB 2025-02-15 11:26:54,496 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 11:26:54,497 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:26:54,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:26:54,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:26:54,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:26:54,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:26:54,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27115.62 MB 2025-02-15 11:26:54,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35533.67 MB 2025-02-15 11:26:54,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.05 MB 2025-02-15 11:26:54,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41636.86 MB 2025-02-15 11:26:54,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45822.77 MB 2025-02-15 11:26:54,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 11:26:54,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35533.67 MB 2025-02-15 11:26:54,664 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 11:26:54,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:54,665 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:26:54,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:54,666 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:26:54,671 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:26:54,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:26:54,672 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:26:54,672 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:27:37,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:37,951 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:27:37,956 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:27:37,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:37,960 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:27:37,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:37,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:27:40,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:27:40,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:27:40,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.71 seconds 2025-02-15 11:27:40,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:40,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-15 11:27:40,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-15 11:27:40,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-15 11:27:40,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54190.41 MB 2025-02-15 11:27:40,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18601.74 MB 2025-02-15 11:27:40,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35588.67 MB 2025-02-15 11:27:40,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23652.54 MB 2025-02-15 11:27:40,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:27:40,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:27:40,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:27:40,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:40,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-15 11:27:40,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15003.98 MB 2025-02-15 11:27:40,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.04 MB 2025-02-15 11:27:40,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18601.74 MB 2025-02-15 11:27:40,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18601.74 MB 2025-02-15 11:27:40,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:27:40,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17058.42 MB 2025-02-15 11:27:41,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:27:41,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:27:41,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 11:27:41,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15003.98 MB 2025-02-15 11:27:41,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15217.65 MB 2025-02-15 11:27:41,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 11:27:41,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18601.74 MB 2025-02-15 11:27:41,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17658.02 MB 2025-02-15 11:27:41,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 11:27:41,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19174.67 MB 2025-02-15 11:27:41,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:27:41,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:27:41,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:27:41,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-15 11:27:41,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.94 MB 2025-02-15 11:27:41,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 11:27:41,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 11:27:41,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17658.02 MB 2025-02-15 11:27:41,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:27:41,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16548.46 MB 2025-02-15 11:27:41,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:27:41,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:27:41,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:27:41,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.94 MB 2025-02-15 11:27:41,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-15 11:27:41,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 11:27:41,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 11:27:41,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20138.95 MB 2025-02-15 11:27:41,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-15 11:27:41,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.77 MB 2025-02-15 11:27:41,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:27:41,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:27:41,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:27:41,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-15 11:27:41,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-15 11:27:41,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 11:27:41,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17658.02 MB 2025-02-15 11:27:41,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20138.95 MB 2025-02-15 11:27:41,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-15 11:27:41,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.77 MB 2025-02-15 11:27:41,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:27:41,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:27:41,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:27:41,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17497.57 MB 2025-02-15 11:27:41,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17807.21 MB 2025-02-15 11:27:41,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-15 11:27:41,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20138.95 MB 2025-02-15 11:27:41,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-15 11:27:41,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-15 11:27:41,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18100.64 MB 2025-02-15 11:27:41,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:27:41,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:27:41,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:27:41,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17973.40 MB 2025-02-15 11:27:41,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18200.87 MB 2025-02-15 11:27:41,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.46 MB 2025-02-15 11:27:41,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-15 11:27:41,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-15 11:27:41,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:27:41,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18226.53 MB 2025-02-15 11:27:41,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:27:41,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:27:41,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.70 seconds 2025-02-15 11:27:41,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-15 11:27:41,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18401.94 MB 2025-02-15 11:27:41,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4827.00 MB 2025-02-15 11:27:41,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54190.41 MB 2025-02-15 11:27:41,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-15 11:27:41,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33883.68 MB 2025-02-15 11:27:41,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18401.94 MB 2025-02-15 11:27:41,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:27:41,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:27:41,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:27:41,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18401.94 MB 2025-02-15 11:27:41,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17452.07 MB 2025-02-15 11:27:41,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -949.87 MB 2025-02-15 11:27:41,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-15 11:27:41,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-15 11:27:41,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:27:41,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19205.67 MB 2025-02-15 11:27:41,947 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:27:41,947 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:27:41,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:27:41,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:27:41,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:27:41,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:27:41,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17452.07 MB 2025-02-15 11:27:41,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25891.09 MB 2025-02-15 11:27:41,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:27:41,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-15 11:27:41,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30796.68 MB 2025-02-15 11:27:41,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:27:41,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25891.09 MB 2025-02-15 11:27:42,117 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:27:42,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:42,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:27:42,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:42,119 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:27:42,124 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:27:42,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:27:42,125 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:27:42,125 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:28:49,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:28:49,723 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:28:49,728 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:28:49,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:28:49,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1047, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:28:49,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:28:49,734 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1047, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:29:05,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:29:05,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:29:05,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.08 seconds 2025-02-15 11:29:05,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:05,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.37 MB 2025-02-15 11:29:05,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23970.04 MB 2025-02-15 11:29:05,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3705.67 MB 2025-02-15 11:29:05,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43381.69 MB 2025-02-15 11:29:05,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26476.54 MB 2025-02-15 11:29:05,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16905.14 MB 2025-02-15 11:29:05,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32907.44 MB 2025-02-15 11:29:05,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:29:05,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:29:05,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:29:05,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:05,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23970.04 MB 2025-02-15 11:29:05,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21221.93 MB 2025-02-15 11:29:05,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2748.11 MB 2025-02-15 11:29:05,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26476.54 MB 2025-02-15 11:29:05,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32747.03 MB 2025-02-15 11:29:05,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6270.48 MB 2025-02-15 11:29:05,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32340.64 MB 2025-02-15 11:29:07,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:29:07,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:29:07,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:29:07,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:07,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21221.93 MB 2025-02-15 11:29:07,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21752.77 MB 2025-02-15 11:29:07,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:29:07,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32747.03 MB 2025-02-15 11:29:07,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24895.29 MB 2025-02-15 11:29:07,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7851.74 MB 2025-02-15 11:29:07,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25732.35 MB 2025-02-15 11:29:07,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:29:07,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:29:07,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:29:07,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:07,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21752.77 MB 2025-02-15 11:29:07,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23642.30 MB 2025-02-15 11:29:07,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:29:07,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 11:29:07,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26782.73 MB 2025-02-15 11:29:07,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 11:29:07,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25059.73 MB 2025-02-15 11:29:08,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:29:08,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:29:08,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:29:08,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23642.30 MB 2025-02-15 11:29:08,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25884.16 MB 2025-02-15 11:29:08,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:29:08,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26782.73 MB 2025-02-15 11:29:08,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 11:29:08,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:29:08,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31428.44 MB 2025-02-15 11:29:08,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:29:08,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:29:08,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:29:08,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21752.77 MB 2025-02-15 11:29:08,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25884.16 MB 2025-02-15 11:29:08,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:29:08,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 11:29:08,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 11:29:08,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 11:29:08,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31428.44 MB 2025-02-15 11:29:08,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:29:08,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:29:08,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:29:08,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27417.70 MB 2025-02-15 11:29:08,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28184.70 MB 2025-02-15 11:29:08,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:29:08,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33388.76 MB 2025-02-15 11:29:08,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 11:29:08,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:29:08,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.49 MB 2025-02-15 11:29:08,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:29:08,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:29:08,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:29:08,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28597.59 MB 2025-02-15 11:29:08,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28829.19 MB 2025-02-15 11:29:08,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.60 MB 2025-02-15 11:29:08,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 11:29:08,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 11:29:08,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:29:08,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29019.94 MB 2025-02-15 11:29:08,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:29:08,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:29:08,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.51 seconds 2025-02-15 11:29:08,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16616.54 MB 2025-02-15 11:29:08,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29030.27 MB 2025-02-15 11:29:08,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12413.73 MB 2025-02-15 11:29:08,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43381.69 MB 2025-02-15 11:29:08,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 11:29:08,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9575.60 MB 2025-02-15 11:29:08,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29030.27 MB 2025-02-15 11:29:08,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:29:08,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:29:08,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:29:08,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29030.27 MB 2025-02-15 11:29:08,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21620.93 MB 2025-02-15 11:29:08,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7409.34 MB 2025-02-15 11:29:08,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 11:29:08,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 11:29:08,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:29:08,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.93 MB 2025-02-15 11:29:08,533 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:29:08,533 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:29:08,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:29:08,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:29:08,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:29:08,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:08,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21620.93 MB 2025-02-15 11:29:08,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30059.95 MB 2025-02-15 11:29:08,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:29:08,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 11:29:08,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42196.80 MB 2025-02-15 11:29:08,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:29:08,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30059.95 MB 2025-02-15 11:29:08,702 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:29:08,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:08,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:29:08,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:08,704 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:29:08,709 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:29:08,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:08,710 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:29:08,710 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 11:29:17,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:17,337 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:29:17,343 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:29:17,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:17,347 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:29:17,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:17,348 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:29:43,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:29:43,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:29:43,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.36 seconds 2025-02-15 11:29:43,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:43,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24758.83 MB 2025-02-15 11:29:43,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30746.73 MB 2025-02-15 11:29:43,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5987.89 MB 2025-02-15 11:29:43,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54781.80 MB 2025-02-15 11:29:43,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39246.10 MB 2025-02-15 11:29:43,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15535.70 MB 2025-02-15 11:29:43,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39666.02 MB 2025-02-15 11:29:43,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:29:43,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:29:43,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:29:43,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:43,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30746.73 MB 2025-02-15 11:29:43,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24574.03 MB 2025-02-15 11:29:43,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6172.70 MB 2025-02-15 11:29:43,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39246.10 MB 2025-02-15 11:29:43,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52282.00 MB 2025-02-15 11:29:43,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13035.90 MB 2025-02-15 11:29:43,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48219.23 MB 2025-02-15 11:29:45,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:29:45,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:29:45,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:29:45,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:45,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24574.03 MB 2025-02-15 11:29:45,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25104.87 MB 2025-02-15 11:29:45,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:29:45,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52282.00 MB 2025-02-15 11:29:45,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:29:45,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17609.79 MB 2025-02-15 11:29:45,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29083.42 MB 2025-02-15 11:29:45,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:29:45,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:29:45,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:29:45,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:45,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25104.87 MB 2025-02-15 11:29:45,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26994.41 MB 2025-02-15 11:29:45,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:29:45,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:29:45,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:29:45,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:29:45,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28411.84 MB 2025-02-15 11:29:46,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:29:46,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:29:46,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:29:46,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26994.41 MB 2025-02-15 11:29:46,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29236.26 MB 2025-02-15 11:29:46,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:29:46,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:29:46,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37031.51 MB 2025-02-15 11:29:46,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 11:29:46,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.54 MB 2025-02-15 11:29:46,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:29:46,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:29:46,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:29:46,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25104.87 MB 2025-02-15 11:29:46,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29236.26 MB 2025-02-15 11:29:46,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:29:46,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:29:46,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37031.51 MB 2025-02-15 11:29:46,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 11:29:46,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.54 MB 2025-02-15 11:29:46,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:29:46,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:29:46,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:29:46,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30769.80 MB 2025-02-15 11:29:46,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31536.81 MB 2025-02-15 11:29:46,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:29:46,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37031.51 MB 2025-02-15 11:29:46,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-15 11:29:46,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:29:46,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32244.60 MB 2025-02-15 11:29:46,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:29:46,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:29:46,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:29:46,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31949.70 MB 2025-02-15 11:29:46,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32178.75 MB 2025-02-15 11:29:46,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 11:29:46,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37446.75 MB 2025-02-15 11:29:46,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-15 11:29:46,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:29:46,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32404.16 MB 2025-02-15 11:29:46,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:29:46,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:29:46,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.85 seconds 2025-02-15 11:29:46,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18863.77 MB 2025-02-15 11:29:46,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32379.61 MB 2025-02-15 11:29:46,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13515.84 MB 2025-02-15 11:29:46,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54781.80 MB 2025-02-15 11:29:46,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-15 11:29:46,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17335.06 MB 2025-02-15 11:29:46,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32404.16 MB 2025-02-15 11:29:46,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:29:46,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:29:46,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:29:46,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32379.61 MB 2025-02-15 11:29:46,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.10 MB 2025-02-15 11:29:46,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8519.51 MB 2025-02-15 11:29:46,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37446.75 MB 2025-02-15 11:29:46,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-15 11:29:46,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:29:46,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34884.51 MB 2025-02-15 11:29:46,491 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 11:29:46,492 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:29:46,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:29:46,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:29:46,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:29:46,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:29:46,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23860.10 MB 2025-02-15 11:29:46,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32276.70 MB 2025-02-15 11:29:46,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 11:29:46,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37446.75 MB 2025-02-15 11:29:46,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45814.38 MB 2025-02-15 11:29:46,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 11:29:46,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32276.70 MB 2025-02-15 11:29:46,661 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 11:29:46,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:46,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:29:46,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:46,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:29:46,668 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:29:46,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:29:46,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:29:46,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:30:47,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:47,618 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:30:47,623 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:30:47,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:47,628 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:30:47,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:47,629 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:30:50,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:30:50,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:30:50,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.56 seconds 2025-02-15 11:30:50,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:50,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14118.45 MB 2025-02-15 11:30:50,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.38 MB 2025-02-15 11:30:50,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-15 11:30:50,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54182.02 MB 2025-02-15 11:30:50,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17895.00 MB 2025-02-15 11:30:50,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36287.02 MB 2025-02-15 11:30:50,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23589.82 MB 2025-02-15 11:30:50,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:30:50,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:30:50,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:30:50,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:50,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.38 MB 2025-02-15 11:30:50,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14957.85 MB 2025-02-15 11:30:50,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.47 MB 2025-02-15 11:30:50,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17895.00 MB 2025-02-15 11:30:50,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 11:30:50,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-15 11:30:50,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16964.51 MB 2025-02-15 11:30:50,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:30:50,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:30:50,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 11:30:50,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:50,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14957.85 MB 2025-02-15 11:30:50,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15171.52 MB 2025-02-15 11:30:50,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 11:30:50,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 11:30:50,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 11:30:50,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:30:50,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19128.54 MB 2025-02-15 11:30:50,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:30:50,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:30:50,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:30:50,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:50,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15171.45 MB 2025-02-15 11:30:50,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15931.80 MB 2025-02-15 11:30:50,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 11:30:50,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 11:30:50,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 11:30:50,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:30:50,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16502.32 MB 2025-02-15 11:30:51,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:30:51,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:30:51,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:30:51,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15931.80 MB 2025-02-15 11:30:51,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16834.19 MB 2025-02-15 11:30:51,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 11:30:51,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 11:30:51,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20373.83 MB 2025-02-15 11:30:51,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-15 11:30:51,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19065.72 MB 2025-02-15 11:30:51,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:30:51,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:30:51,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:30:51,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15171.45 MB 2025-02-15 11:30:51,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16834.19 MB 2025-02-15 11:30:51,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 11:30:51,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 11:30:51,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20373.83 MB 2025-02-15 11:30:51,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-15 11:30:51,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19065.72 MB 2025-02-15 11:30:51,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:30:51,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:30:51,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:30:51,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17451.44 MB 2025-02-15 11:30:51,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17760.16 MB 2025-02-15 11:30:51,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 11:30:51,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20373.83 MB 2025-02-15 11:30:51,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20539.51 MB 2025-02-15 11:30:51,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 11:30:51,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18052.96 MB 2025-02-15 11:30:51,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:30:51,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:30:51,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:30:51,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17926.35 MB 2025-02-15 11:30:51,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18151.56 MB 2025-02-15 11:30:51,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.21 MB 2025-02-15 11:30:51,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20539.51 MB 2025-02-15 11:30:51,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20539.51 MB 2025-02-15 11:30:51,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:30:51,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18166.95 MB 2025-02-15 11:30:51,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:30:51,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:30:51,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.53 seconds 2025-02-15 11:30:51,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13543.58 MB 2025-02-15 11:30:51,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18352.46 MB 2025-02-15 11:30:51,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4808.89 MB 2025-02-15 11:30:51,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54182.02 MB 2025-02-15 11:30:51,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20539.51 MB 2025-02-15 11:30:51,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33642.51 MB 2025-02-15 11:30:51,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18352.46 MB 2025-02-15 11:30:51,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:30:51,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:30:51,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:30:51,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18352.46 MB 2025-02-15 11:30:51,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17417.39 MB 2025-02-15 11:30:51,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -935.07 MB 2025-02-15 11:30:51,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20539.51 MB 2025-02-15 11:30:51,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20539.51 MB 2025-02-15 11:30:51,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:30:51,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19155.51 MB 2025-02-15 11:30:51,447 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 11:30:51,448 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:30:51,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:30:51,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:30:51,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:30:51,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:30:51,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17417.39 MB 2025-02-15 11:30:51,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25848.86 MB 2025-02-15 11:30:51,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 11:30:51,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20539.51 MB 2025-02-15 11:30:51,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31021.07 MB 2025-02-15 11:30:51,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 11:30:51,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25848.86 MB 2025-02-15 11:30:51,618 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 11:30:51,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:51,619 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:30:51,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:51,620 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:30:51,625 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:30:51,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:30:51,626 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:30:51,626 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:31:49,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:31:49,584 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:31:49,589 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:31:49,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:31:49,593 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1498, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:31:49,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:31:49,594 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1498, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:32:12,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:32:12,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:32:12,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.04 seconds 2025-02-15 11:32:12,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:12,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23407.01 MB 2025-02-15 11:32:12,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28708.61 MB 2025-02-15 11:32:12,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5301.60 MB 2025-02-15 11:32:12,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39405.49 MB 2025-02-15 11:32:12,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38549.85 MB 2025-02-15 11:32:12,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -855.64 MB 2025-02-15 11:32:12,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37634.72 MB 2025-02-15 11:32:12,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:32:12,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:32:12,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:32:12,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:12,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28708.61 MB 2025-02-15 11:32:12,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23565.49 MB 2025-02-15 11:32:12,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5143.12 MB 2025-02-15 11:32:12,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38549.85 MB 2025-02-15 11:32:12,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48077.21 MB 2025-02-15 11:32:12,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9527.36 MB 2025-02-15 11:32:12,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43032.59 MB 2025-02-15 11:32:14,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:32:14,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:32:14,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:32:14,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:14,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23565.49 MB 2025-02-15 11:32:14,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24096.33 MB 2025-02-15 11:32:14,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:32:14,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48077.21 MB 2025-02-15 11:32:14,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 11:32:14,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19021.17 MB 2025-02-15 11:32:14,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28074.87 MB 2025-02-15 11:32:14,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:32:14,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:32:14,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:32:14,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:14,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24096.33 MB 2025-02-15 11:32:14,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.86 MB 2025-02-15 11:32:14,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:32:14,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 11:32:14,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29056.04 MB 2025-02-15 11:32:14,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:32:14,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27403.29 MB 2025-02-15 11:32:15,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:32:15,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:32:15,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-15 11:32:15,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25985.86 MB 2025-02-15 11:32:15,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28227.72 MB 2025-02-15 11:32:15,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:32:15,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 11:32:15,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36370.91 MB 2025-02-15 11:32:15,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:32:15,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.00 MB 2025-02-15 11:32:15,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:32:15,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:32:15,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.37 seconds 2025-02-15 11:32:15,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24096.33 MB 2025-02-15 11:32:15,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28227.72 MB 2025-02-15 11:32:15,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:32:15,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29056.04 MB 2025-02-15 11:32:15,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36370.91 MB 2025-02-15 11:32:15,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 11:32:15,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.00 MB 2025-02-15 11:32:15,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:32:15,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:32:15,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:32:15,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29761.26 MB 2025-02-15 11:32:15,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30528.26 MB 2025-02-15 11:32:15,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:32:15,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36370.91 MB 2025-02-15 11:32:15,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:32:15,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:32:15,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31236.05 MB 2025-02-15 11:32:15,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:32:15,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:32:15,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:32:15,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30941.15 MB 2025-02-15 11:32:15,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31171.07 MB 2025-02-15 11:32:15,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.92 MB 2025-02-15 11:32:15,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:32:15,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:32:15,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:32:15,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31369.27 MB 2025-02-15 11:32:15,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:32:15,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:32:15,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.61 seconds 2025-02-15 11:32:15,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18187.86 MB 2025-02-15 11:32:15,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31372.14 MB 2025-02-15 11:32:15,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13184.28 MB 2025-02-15 11:32:15,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39405.49 MB 2025-02-15 11:32:15,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:32:15,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2619.34 MB 2025-02-15 11:32:15,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31372.14 MB 2025-02-15 11:32:15,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:32:15,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:32:15,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:32:15,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31372.14 MB 2025-02-15 11:32:15,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23192.25 MB 2025-02-15 11:32:15,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8179.89 MB 2025-02-15 11:32:15,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:32:15,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36786.14 MB 2025-02-15 11:32:15,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:32:15,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31372.14 MB 2025-02-15 11:32:15,495 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:32:15,496 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:32:15,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:32:15,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:32:15,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:32:15,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:32:15,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.25 MB 2025-02-15 11:32:15,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31631.27 MB 2025-02-15 11:32:15,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:32:15,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36786.14 MB 2025-02-15 11:32:15,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-15 11:32:15,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:32:15,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31631.27 MB 2025-02-15 11:32:15,661 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:32:15,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:15,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:32:15,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:15,664 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:32:15,668 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:32:15,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:15,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:32:15,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:32:43,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:43,506 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:32:43,511 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:32:43,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:43,514 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:32:43,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:32:43,515 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:33:03,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:33:03,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:33:03,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.67 seconds 2025-02-15 11:33:03,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:03,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-15 11:33:03,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-15 11:33:03,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-15 11:33:03,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57761.86 MB 2025-02-15 11:33:03,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37727.76 MB 2025-02-15 11:33:03,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20034.09 MB 2025-02-15 11:33:03,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-15 11:33:03,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:33:03,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:33:03,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:33:03,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:03,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-15 11:33:03,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-15 11:33:03,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-15 11:33:03,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37727.76 MB 2025-02-15 11:33:03,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46544.19 MB 2025-02-15 11:33:03,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8816.43 MB 2025-02-15 11:33:03,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39518.09 MB 2025-02-15 11:33:05,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:33:05,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:33:05,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 11:33:05,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-15 11:33:05,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-15 11:33:05,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:33:05,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46544.19 MB 2025-02-15 11:33:05,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33250.34 MB 2025-02-15 11:33:05,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13293.85 MB 2025-02-15 11:33:05,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.58 MB 2025-02-15 11:33:05,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:33:05,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:33:05,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:33:05,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-15 11:33:05,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-15 11:33:05,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:33:05,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:33:05,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33250.34 MB 2025-02-15 11:33:05,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:33:05,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-15 11:33:05,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:33:05,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:33:05,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:33:05,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-15 11:33:05,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-15 11:33:05,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:33:05,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:33:05,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33722.20 MB 2025-02-15 11:33:05,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 11:33:05,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-15 11:33:05,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:33:05,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:33:05,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:33:05,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-15 11:33:05,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-15 11:33:05,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:33:05,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:33:05,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33722.20 MB 2025-02-15 11:33:05,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 11:33:05,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-15 11:33:05,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:33:05,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:33:05,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:33:05,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-15 11:33:05,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-15 11:33:05,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:33:05,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33722.20 MB 2025-02-15 11:33:05,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34137.44 MB 2025-02-15 11:33:05,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:33:05,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-15 11:33:05,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:33:05,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:33:05,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:33:05,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-15 11:33:05,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.96 MB 2025-02-15 11:33:05,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-15 11:33:05,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34137.44 MB 2025-02-15 11:33:05,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34137.44 MB 2025-02-15 11:33:05,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:33:05,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30199.28 MB 2025-02-15 11:33:05,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:33:05,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:33:05,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.09 seconds 2025-02-15 11:33:05,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-15 11:33:05,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30159.99 MB 2025-02-15 11:33:05,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.92 MB 2025-02-15 11:33:05,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57761.86 MB 2025-02-15 11:33:05,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34137.44 MB 2025-02-15 11:33:05,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23624.42 MB 2025-02-15 11:33:05,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30199.28 MB 2025-02-15 11:33:05,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:33:05,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:33:05,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:33:05,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30159.99 MB 2025-02-15 11:33:05,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22379.70 MB 2025-02-15 11:33:05,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7780.29 MB 2025-02-15 11:33:05,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34137.44 MB 2025-02-15 11:33:05,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34137.44 MB 2025-02-15 11:33:05,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:33:05,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32671.04 MB 2025-02-15 11:33:05,889 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 11:33:05,889 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:33:05,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:33:05,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:33:05,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:33:05,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:33:05,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22379.70 MB 2025-02-15 11:33:05,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30817.17 MB 2025-02-15 11:33:05,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 11:33:05,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34137.44 MB 2025-02-15 11:33:05,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42526.05 MB 2025-02-15 11:33:05,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 11:33:05,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30817.17 MB 2025-02-15 11:33:06,056 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 11:33:06,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:33:06,058 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:33:06,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:33:06,059 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:33:06,063 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:33:06,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:33:06,064 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:33:06,064 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:34:27,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:27,090 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:34:27,096 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:34:27,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:27,099 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 716, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:34:27,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:27,104 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 716, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:34:38,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:34:38,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:34:38,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.04 seconds 2025-02-15 11:34:38,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:38,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17957.91 MB 2025-02-15 11:34:38,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20491.79 MB 2025-02-15 11:34:38,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2533.88 MB 2025-02-15 11:34:38,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50914.66 MB 2025-02-15 11:34:38,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24863.83 MB 2025-02-15 11:34:38,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26050.82 MB 2025-02-15 11:34:38,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29467.71 MB 2025-02-15 11:34:38,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:34:38,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:34:38,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 11:34:38,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:38,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20491.79 MB 2025-02-15 11:34:38,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19500.11 MB 2025-02-15 11:34:38,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -991.68 MB 2025-02-15 11:34:38,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24863.83 MB 2025-02-15 11:34:38,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31113.35 MB 2025-02-15 11:34:38,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6249.51 MB 2025-02-15 11:34:38,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29550.51 MB 2025-02-15 11:34:40,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:34:40,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:34:40,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 11:34:40,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19500.11 MB 2025-02-15 11:34:40,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20030.95 MB 2025-02-15 11:34:40,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:34:40,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31113.35 MB 2025-02-15 11:34:40,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26279.41 MB 2025-02-15 11:34:40,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4833.94 MB 2025-02-15 11:34:40,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.50 MB 2025-02-15 11:34:40,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:34:40,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:34:40,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:34:40,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20030.95 MB 2025-02-15 11:34:40,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21920.49 MB 2025-02-15 11:34:40,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:34:40,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26279.41 MB 2025-02-15 11:34:40,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26279.41 MB 2025-02-15 11:34:40,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:34:40,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23337.92 MB 2025-02-15 11:34:40,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:34:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:34:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:34:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21920.49 MB 2025-02-15 11:34:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24162.34 MB 2025-02-15 11:34:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:34:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26279.41 MB 2025-02-15 11:34:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 11:34:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 11:34:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.63 MB 2025-02-15 11:34:40,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:34:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:34:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:34:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20030.95 MB 2025-02-15 11:34:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24162.34 MB 2025-02-15 11:34:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:34:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26279.41 MB 2025-02-15 11:34:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31943.82 MB 2025-02-15 11:34:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 11:34:40,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.63 MB 2025-02-15 11:34:40,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:34:40,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:34:40,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:34:40,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25695.89 MB 2025-02-15 11:34:40,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26462.89 MB 2025-02-15 11:34:40,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:34:40,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31943.82 MB 2025-02-15 11:34:40,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32359.06 MB 2025-02-15 11:34:40,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:34:40,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27170.68 MB 2025-02-15 11:34:40,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:34:40,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:34:40,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:34:40,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26875.78 MB 2025-02-15 11:34:40,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27105.18 MB 2025-02-15 11:34:40,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.40 MB 2025-02-15 11:34:40,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32359.06 MB 2025-02-15 11:34:40,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32359.06 MB 2025-02-15 11:34:40,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:34:40,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27302.33 MB 2025-02-15 11:34:40,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:34:40,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:34:40,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.43 seconds 2025-02-15 11:34:40,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15463.31 MB 2025-02-15 11:34:40,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27306.25 MB 2025-02-15 11:34:40,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11842.95 MB 2025-02-15 11:34:40,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50914.66 MB 2025-02-15 11:34:40,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32359.06 MB 2025-02-15 11:34:40,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18555.60 MB 2025-02-15 11:34:40,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27306.25 MB 2025-02-15 11:34:40,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:34:40,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:34:40,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:34:40,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27306.25 MB 2025-02-15 11:34:40,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20467.70 MB 2025-02-15 11:34:40,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6838.56 MB 2025-02-15 11:34:40,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32359.06 MB 2025-02-15 11:34:40,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32359.06 MB 2025-02-15 11:34:40,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:34:40,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29817.92 MB 2025-02-15 11:34:40,823 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:34:40,823 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:34:40,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:34:40,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:34:40,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:34:40,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:34:40,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20467.70 MB 2025-02-15 11:34:40,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.72 MB 2025-02-15 11:34:40,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:34:40,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32359.06 MB 2025-02-15 11:34:40,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40749.76 MB 2025-02-15 11:34:40,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:34:40,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28906.72 MB 2025-02-15 11:34:40,994 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:34:40,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:40,995 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:34:40,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:40,996 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:34:41,001 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:34:41,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:34:41,002 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:34:41,002 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:35:23,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:23,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:35:23,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:35:23,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:23,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2007, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:35:23,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:23,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2007, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:35:55,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:35:55,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:35:55,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.11 seconds 2025-02-15 11:35:55,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:55,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26953.80 MB 2025-02-15 11:35:55,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34056.86 MB 2025-02-15 11:35:55,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7103.05 MB 2025-02-15 11:35:55,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53334.77 MB 2025-02-15 11:35:55,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40355.50 MB 2025-02-15 11:35:55,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12979.27 MB 2025-02-15 11:35:55,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42993.45 MB 2025-02-15 11:35:55,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:35:55,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:35:55,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:35:55,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:55,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34056.86 MB 2025-02-15 11:35:55,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26211.62 MB 2025-02-15 11:35:55,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7845.24 MB 2025-02-15 11:35:55,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40355.50 MB 2025-02-15 11:35:55,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55293.51 MB 2025-02-15 11:35:55,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14938.01 MB 2025-02-15 11:35:55,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54392.11 MB 2025-02-15 11:35:57,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:35:57,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:35:57,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:35:57,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26211.62 MB 2025-02-15 11:35:57,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.46 MB 2025-02-15 11:35:57,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:35:57,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55293.51 MB 2025-02-15 11:35:57,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30475.81 MB 2025-02-15 11:35:57,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24817.70 MB 2025-02-15 11:35:57,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30722.66 MB 2025-02-15 11:35:57,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:35:57,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:35:57,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:35:57,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.46 MB 2025-02-15 11:35:57,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28631.99 MB 2025-02-15 11:35:57,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:35:57,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30475.81 MB 2025-02-15 11:35:57,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32363.25 MB 2025-02-15 11:35:57,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 11:35:57,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30049.42 MB 2025-02-15 11:35:57,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:35:57,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:35:57,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:35:57,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28631.99 MB 2025-02-15 11:35:57,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30873.85 MB 2025-02-15 11:35:57,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:35:57,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32363.25 MB 2025-02-15 11:35:57,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38497.42 MB 2025-02-15 11:35:57,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:35:57,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36418.13 MB 2025-02-15 11:35:57,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:35:57,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:35:57,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:35:57,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.46 MB 2025-02-15 11:35:57,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30873.85 MB 2025-02-15 11:35:57,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:35:57,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30475.81 MB 2025-02-15 11:35:57,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38497.42 MB 2025-02-15 11:35:57,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 11:35:57,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36418.13 MB 2025-02-15 11:35:57,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:35:57,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:35:57,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:35:57,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.39 MB 2025-02-15 11:35:57,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33174.39 MB 2025-02-15 11:35:57,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:35:57,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38497.42 MB 2025-02-15 11:35:57,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 11:35:57,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:35:57,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33882.18 MB 2025-02-15 11:35:57,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:35:57,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:35:57,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:35:57,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33587.28 MB 2025-02-15 11:35:57,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33815.24 MB 2025-02-15 11:35:57,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-15 11:35:57,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38912.66 MB 2025-02-15 11:35:57,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 11:35:57,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:35:57,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34038.32 MB 2025-02-15 11:35:57,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:35:57,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:35:57,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.65 seconds 2025-02-15 11:35:57,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19961.25 MB 2025-02-15 11:35:57,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34016.09 MB 2025-02-15 11:35:57,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14054.83 MB 2025-02-15 11:35:57,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53334.77 MB 2025-02-15 11:35:57,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 11:35:57,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14422.11 MB 2025-02-15 11:35:57,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34038.32 MB 2025-02-15 11:35:57,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:35:57,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:35:57,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:35:57,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34016.09 MB 2025-02-15 11:35:57,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24947.96 MB 2025-02-15 11:35:57,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9068.12 MB 2025-02-15 11:35:57,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38912.66 MB 2025-02-15 11:35:57,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38912.66 MB 2025-02-15 11:35:57,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:35:57,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36512.70 MB 2025-02-15 11:35:57,849 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 11:35:57,849 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 11:35:57,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:35:57,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:35:57,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:35:57,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:35:57,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24947.96 MB 2025-02-15 11:35:57,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33336.07 MB 2025-02-15 11:35:57,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.11 MB 2025-02-15 11:35:57,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38912.66 MB 2025-02-15 11:35:57,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43083.89 MB 2025-02-15 11:35:57,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 11:35:57,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33336.07 MB 2025-02-15 11:35:58,014 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 11:35:58,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:58,015 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:35:58,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:58,016 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:35:58,021 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:35:58,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:35:58,022 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:35:58,022 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 11:36:10,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:10,806 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:36:10,812 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:36:10,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:10,817 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1015, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:36:10,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:10,819 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1015, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:36:26,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:36:26,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:36:26,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.94 seconds 2025-02-15 11:36:26,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:26,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20041.39 MB 2025-02-15 11:36:26,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23633.81 MB 2025-02-15 11:36:26,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3592.42 MB 2025-02-15 11:36:26,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51422.17 MB 2025-02-15 11:36:26,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28410.12 MB 2025-02-15 11:36:26,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23012.05 MB 2025-02-15 11:36:26,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32457.16 MB 2025-02-15 11:36:26,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:36:26,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:36:26,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:36:26,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:26,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23633.81 MB 2025-02-15 11:36:26,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21055.57 MB 2025-02-15 11:36:26,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2578.24 MB 2025-02-15 11:36:26,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28410.12 MB 2025-02-15 11:36:26,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37232.84 MB 2025-02-15 11:36:26,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8822.72 MB 2025-02-15 11:36:26,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34552.59 MB 2025-02-15 11:36:28,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:36:28,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:36:28,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:36:28,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:28,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21055.57 MB 2025-02-15 11:36:28,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21586.41 MB 2025-02-15 11:36:28,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:36:28,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37232.84 MB 2025-02-15 11:36:28,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26942.11 MB 2025-02-15 11:36:28,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10290.72 MB 2025-02-15 11:36:28,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25564.96 MB 2025-02-15 11:36:28,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:36:28,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:36:28,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:36:28,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:28,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21586.41 MB 2025-02-15 11:36:28,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23475.94 MB 2025-02-15 11:36:28,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:36:28,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26942.11 MB 2025-02-15 11:36:28,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26942.11 MB 2025-02-15 11:36:28,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:36:28,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24893.37 MB 2025-02-15 11:36:29,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:36:29,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:36:29,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:36:29,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23475.94 MB 2025-02-15 11:36:29,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.80 MB 2025-02-15 11:36:29,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:36:29,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26942.11 MB 2025-02-15 11:36:29,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33076.28 MB 2025-02-15 11:36:29,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:36:29,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31262.08 MB 2025-02-15 11:36:29,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:36:29,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:36:29,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:36:29,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21586.41 MB 2025-02-15 11:36:29,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.80 MB 2025-02-15 11:36:29,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:36:29,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26942.11 MB 2025-02-15 11:36:29,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33076.28 MB 2025-02-15 11:36:29,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:36:29,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31262.08 MB 2025-02-15 11:36:29,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:36:29,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:36:29,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:36:29,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27251.34 MB 2025-02-15 11:36:29,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28018.34 MB 2025-02-15 11:36:29,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:36:29,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33076.28 MB 2025-02-15 11:36:29,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-15 11:36:29,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:36:29,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28726.13 MB 2025-02-15 11:36:29,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:36:29,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:36:29,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:36:29,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28431.23 MB 2025-02-15 11:36:29,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28660.02 MB 2025-02-15 11:36:29,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 11:36:29,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33491.52 MB 2025-02-15 11:36:29,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-15 11:36:29,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:36:29,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28880.79 MB 2025-02-15 11:36:29,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:36:29,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:36:29,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.40 seconds 2025-02-15 11:36:29,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16505.05 MB 2025-02-15 11:36:29,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28860.87 MB 2025-02-15 11:36:29,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12355.83 MB 2025-02-15 11:36:29,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51422.17 MB 2025-02-15 11:36:29,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-15 11:36:29,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17930.65 MB 2025-02-15 11:36:29,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28880.79 MB 2025-02-15 11:36:29,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:36:29,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:36:29,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:36:29,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28860.87 MB 2025-02-15 11:36:29,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21494.61 MB 2025-02-15 11:36:29,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7366.27 MB 2025-02-15 11:36:29,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33491.52 MB 2025-02-15 11:36:29,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-15 11:36:29,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:36:29,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31359.94 MB 2025-02-15 11:36:29,510 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 11:36:29,511 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:36:29,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:36:29,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:36:29,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:36:29,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:29,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21494.61 MB 2025-02-15 11:36:29,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29891.37 MB 2025-02-15 11:36:29,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-15 11:36:29,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33491.52 MB 2025-02-15 11:36:29,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41840.28 MB 2025-02-15 11:36:29,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-15 11:36:29,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29891.37 MB 2025-02-15 11:36:29,677 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 11:36:29,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:29,679 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:36:29,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:29,680 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:36:29,684 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:36:29,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:29,685 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:36:29,686 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:36:45,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:45,005 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:36:45,013 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:36:45,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:45,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 321, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:36:45,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:45,021 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 321, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:36:50,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:36:50,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:36:50,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.10 seconds 2025-02-15 11:36:50,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:50,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15205.49 MB 2025-02-15 11:36:50,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16341.49 MB 2025-02-15 11:36:50,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1136.00 MB 2025-02-15 11:36:50,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54362.37 MB 2025-02-15 11:36:50,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19075.69 MB 2025-02-15 11:36:50,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35286.68 MB 2025-02-15 11:36:50,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25356.33 MB 2025-02-15 11:36:50,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:36:50,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:36:50,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:36:50,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:50,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16341.49 MB 2025-02-15 11:36:50,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16660.05 MB 2025-02-15 11:36:50,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.56 MB 2025-02-15 11:36:50,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19075.69 MB 2025-02-15 11:36:50,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22613.59 MB 2025-02-15 11:36:50,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3537.90 MB 2025-02-15 11:36:50,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20386.75 MB 2025-02-15 11:36:51,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:36:51,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:36:51,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.39 seconds 2025-02-15 11:36:51,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16660.05 MB 2025-02-15 11:36:51,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.26 MB 2025-02-15 11:36:51,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 382.21 MB 2025-02-15 11:36:51,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22613.59 MB 2025-02-15 11:36:51,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20155.73 MB 2025-02-15 11:36:51,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2457.86 MB 2025-02-15 11:36:51,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21001.65 MB 2025-02-15 11:36:51,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:36:51,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:36:51,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:36:51,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-15 11:36:51,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18403.31 MB 2025-02-15 11:36:51,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1361.05 MB 2025-02-15 11:36:51,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20155.73 MB 2025-02-15 11:36:51,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20835.21 MB 2025-02-15 11:36:51,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 679.48 MB 2025-02-15 11:36:51,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19423.86 MB 2025-02-15 11:36:51,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:36:51,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:36:51,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 11:36:51,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.31 MB 2025-02-15 11:36:51,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-15 11:36:51,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1614.15 MB 2025-02-15 11:36:51,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20835.21 MB 2025-02-15 11:36:51,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25251.81 MB 2025-02-15 11:36:51,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4416.60 MB 2025-02-15 11:36:51,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-15 11:36:51,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:36:51,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:36:51,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:36:51,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-15 11:36:51,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-15 11:36:51,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2975.21 MB 2025-02-15 11:36:51,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20155.73 MB 2025-02-15 11:36:51,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25251.81 MB 2025-02-15 11:36:51,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5096.08 MB 2025-02-15 11:36:51,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-15 11:36:51,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:36:51,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:36:51,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:36:51,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21121.61 MB 2025-02-15 11:36:51,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21673.85 MB 2025-02-15 11:36:51,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.24 MB 2025-02-15 11:36:51,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25251.81 MB 2025-02-15 11:36:51,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25549.60 MB 2025-02-15 11:36:51,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 297.80 MB 2025-02-15 11:36:51,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22183.46 MB 2025-02-15 11:36:51,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:36:51,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:36:51,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:36:51,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21971.14 MB 2025-02-15 11:36:51,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22199.79 MB 2025-02-15 11:36:51,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.65 MB 2025-02-15 11:36:51,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25549.60 MB 2025-02-15 11:36:51,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25549.60 MB 2025-02-15 11:36:51,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:36:51,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22285.82 MB 2025-02-15 11:36:51,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:36:51,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:36:51,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.81 seconds 2025-02-15 11:36:51,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:51,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14087.10 MB 2025-02-15 11:36:51,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22400.84 MB 2025-02-15 11:36:51,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8313.74 MB 2025-02-15 11:36:51,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54362.37 MB 2025-02-15 11:36:51,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25549.60 MB 2025-02-15 11:36:51,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28812.77 MB 2025-02-15 11:36:51,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22400.84 MB 2025-02-15 11:36:52,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:36:52,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:36:52,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:36:52,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:52,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22400.84 MB 2025-02-15 11:36:52,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25414.50 MB 2025-02-15 11:36:52,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3013.66 MB 2025-02-15 11:36:52,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25549.60 MB 2025-02-15 11:36:52,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27160.22 MB 2025-02-15 11:36:52,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1610.61 MB 2025-02-15 11:36:52,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25716.11 MB 2025-02-15 11:36:52,127 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 11:36:52,127 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:36:52,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:36:52,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:36:52,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:36:52,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:36:52,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18562.54 MB 2025-02-15 11:36:52,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27001.38 MB 2025-02-15 11:36:52,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-15 11:36:52,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27160.22 MB 2025-02-15 11:36:52,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37645.98 MB 2025-02-15 11:36:52,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 11:36:52,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27001.38 MB 2025-02-15 11:36:52,295 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 11:36:52,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:52,296 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:36:52,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:52,297 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:36:52,302 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:36:52,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:36:52,303 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:36:52,303 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 11:38:08,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:08,343 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:38:08,352 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:38:08,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:08,359 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:38:08,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:08,361 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:38:12,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:38:12,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:38:12,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.74 seconds 2025-02-15 11:38:12,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:12,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-15 11:38:12,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15427.37 MB 2025-02-15 11:38:12,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-15 11:38:12,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46034.58 MB 2025-02-15 11:38:12,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18842.91 MB 2025-02-15 11:38:12,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27191.67 MB 2025-02-15 11:38:12,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.12 MB 2025-02-15 11:38:12,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:38:12,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:38:12,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:38:12,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:12,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15427.37 MB 2025-02-15 11:38:12,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15828.78 MB 2025-02-15 11:38:12,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 401.42 MB 2025-02-15 11:38:12,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18842.91 MB 2025-02-15 11:38:12,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20499.66 MB 2025-02-15 11:38:12,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1656.75 MB 2025-02-15 11:38:12,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18714.67 MB 2025-02-15 11:38:13,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:38:13,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:38:13,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.16 seconds 2025-02-15 11:38:13,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15828.78 MB 2025-02-15 11:38:13,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16139.33 MB 2025-02-15 11:38:13,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.54 MB 2025-02-15 11:38:13,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20499.66 MB 2025-02-15 11:38:13,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19161.68 MB 2025-02-15 11:38:13,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1337.98 MB 2025-02-15 11:38:13,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20084.41 MB 2025-02-15 11:38:13,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:38:13,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:38:13,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:38:13,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16139.33 MB 2025-02-15 11:38:13,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17244.44 MB 2025-02-15 11:38:13,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1105.11 MB 2025-02-15 11:38:13,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19161.68 MB 2025-02-15 11:38:13,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19715.33 MB 2025-02-15 11:38:13,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 553.65 MB 2025-02-15 11:38:13,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18073.63 MB 2025-02-15 11:38:13,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:38:13,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:38:13,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:38:13,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17244.44 MB 2025-02-15 11:38:13,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18555.95 MB 2025-02-15 11:38:13,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1311.51 MB 2025-02-15 11:38:13,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19715.33 MB 2025-02-15 11:38:13,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 11:38:13,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3598.71 MB 2025-02-15 11:38:13,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21800.11 MB 2025-02-15 11:38:13,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:38:13,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:38:13,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 11:38:13,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16139.33 MB 2025-02-15 11:38:13,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18555.95 MB 2025-02-15 11:38:13,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2416.62 MB 2025-02-15 11:38:13,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19161.68 MB 2025-02-15 11:38:13,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 11:38:13,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4152.36 MB 2025-02-15 11:38:13,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21800.11 MB 2025-02-15 11:38:13,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:38:13,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:38:13,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:38:13,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19453.07 MB 2025-02-15 11:38:13,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19902.55 MB 2025-02-15 11:38:13,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 449.48 MB 2025-02-15 11:38:13,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23314.04 MB 2025-02-15 11:38:13,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23555.21 MB 2025-02-15 11:38:13,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 241.17 MB 2025-02-15 11:38:13,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20316.61 MB 2025-02-15 11:38:13,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:38:13,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:38:13,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:38:13,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20144.10 MB 2025-02-15 11:38:13,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20349.02 MB 2025-02-15 11:38:13,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.92 MB 2025-02-15 11:38:13,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23555.21 MB 2025-02-15 11:38:13,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23559.41 MB 2025-02-15 11:38:13,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 11:38:13,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20417.11 MB 2025-02-15 11:38:13,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:38:13,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:38:13,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.24 seconds 2025-02-15 11:38:13,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13783.98 MB 2025-02-15 11:38:13,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20550.09 MB 2025-02-15 11:38:13,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6766.11 MB 2025-02-15 11:38:13,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46034.58 MB 2025-02-15 11:38:13,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23559.41 MB 2025-02-15 11:38:13,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22475.18 MB 2025-02-15 11:38:13,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20550.09 MB 2025-02-15 11:38:13,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:38:13,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:38:13,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:38:13,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20550.09 MB 2025-02-15 11:38:13,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23564.12 MB 2025-02-15 11:38:13,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 11:38:13,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23559.41 MB 2025-02-15 11:38:13,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25304.24 MB 2025-02-15 11:38:13,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-15 11:38:13,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23865.75 MB 2025-02-15 11:38:13,897 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:38:13,898 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:38:13,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:38:13,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:38:13,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:38:13,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:38:13,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18005.23 MB 2025-02-15 11:38:13,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26444.25 MB 2025-02-15 11:38:13,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:38:13,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25304.24 MB 2025-02-15 11:38:13,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35794.19 MB 2025-02-15 11:38:13,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:38:13,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26444.25 MB 2025-02-15 11:38:14,069 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:38:14,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:14,070 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:38:14,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:14,071 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:38:14,076 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:38:14,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:38:14,077 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:38:14,077 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:39:34,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:39:34,425 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:39:34,431 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:39:34,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:39:34,435 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1487, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:39:34,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:39:34,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1487, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:39:57,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:39:57,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:39:57,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.82 seconds 2025-02-15 11:39:57,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:57,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23330.36 MB 2025-02-15 11:39:57,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28592.77 MB 2025-02-15 11:39:57,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5262.41 MB 2025-02-15 11:39:57,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48379.20 MB 2025-02-15 11:39:57,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38518.39 MB 2025-02-15 11:39:57,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9860.81 MB 2025-02-15 11:39:57,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37558.07 MB 2025-02-15 11:39:57,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:39:57,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:39:57,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:39:57,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:57,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28592.77 MB 2025-02-15 11:39:57,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.30 MB 2025-02-15 11:39:57,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5084.47 MB 2025-02-15 11:39:57,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38518.39 MB 2025-02-15 11:39:57,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47580.18 MB 2025-02-15 11:39:57,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9061.79 MB 2025-02-15 11:39:57,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42219.36 MB 2025-02-15 11:39:59,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:39:59,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:39:59,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:39:59,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.30 MB 2025-02-15 11:39:59,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24039.14 MB 2025-02-15 11:39:59,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:39:59,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47580.18 MB 2025-02-15 11:39:59,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 11:39:59,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18519.95 MB 2025-02-15 11:39:59,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28017.69 MB 2025-02-15 11:39:59,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:39:59,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:39:59,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:39:59,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-15 11:39:59,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.68 MB 2025-02-15 11:39:59,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:39:59,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:39:59,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 11:39:59,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:39:59,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27346.10 MB 2025-02-15 11:39:59,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:39:59,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:39:59,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:39:59,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25928.68 MB 2025-02-15 11:39:59,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-15 11:39:59,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:39:59,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:39:59,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36377.20 MB 2025-02-15 11:39:59,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7316.96 MB 2025-02-15 11:39:59,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-15 11:39:59,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:39:59,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:39:59,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:39:59,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-15 11:39:59,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-15 11:39:59,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:39:59,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:39:59,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36377.20 MB 2025-02-15 11:39:59,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7316.96 MB 2025-02-15 11:39:59,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-15 11:39:59,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:39:59,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:39:59,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:39:59,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29704.07 MB 2025-02-15 11:39:59,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30471.08 MB 2025-02-15 11:39:59,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:39:59,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36377.20 MB 2025-02-15 11:39:59,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 11:39:59,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:39:59,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31178.86 MB 2025-02-15 11:39:59,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:39:59,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:39:59,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:39:59,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30883.96 MB 2025-02-15 11:39:59,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31113.02 MB 2025-02-15 11:39:59,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 11:39:59,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 11:39:59,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 11:39:59,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:39:59,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31315.26 MB 2025-02-15 11:39:59,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:39:59,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:39:59,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.27 seconds 2025-02-15 11:39:59,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18149.53 MB 2025-02-15 11:39:59,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31314.00 MB 2025-02-15 11:39:59,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13164.46 MB 2025-02-15 11:39:59,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48379.20 MB 2025-02-15 11:39:59,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 11:39:59,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11584.67 MB 2025-02-15 11:39:59,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31315.26 MB 2025-02-15 11:39:59,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:39:59,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:39:59,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:39:59,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:39:59,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31314.00 MB 2025-02-15 11:39:59,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23152.40 MB 2025-02-15 11:39:59,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8161.60 MB 2025-02-15 11:39:59,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 11:39:59,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36794.53 MB 2025-02-15 11:39:59,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:39:59,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33824.44 MB 2025-02-15 11:39:59,996 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 11:39:59,996 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:40:00,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:40:00,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:40:00,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:40:00,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:40:00,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23152.40 MB 2025-02-15 11:40:00,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31587.25 MB 2025-02-15 11:40:00,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-15 11:40:00,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36794.53 MB 2025-02-15 11:40:00,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45181.04 MB 2025-02-15 11:40:00,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 11:40:00,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31587.25 MB 2025-02-15 11:40:00,165 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 11:40:00,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:00,166 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:40:00,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:00,167 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:40:00,172 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:40:00,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:00,173 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:40:00,173 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:40:54,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:54,541 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:40:54,547 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:40:54,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:54,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1941, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:40:54,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:40:54,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1941, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:41:24,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:41:24,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:41:24,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.03 seconds 2025-02-15 11:41:24,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:24,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26493.90 MB 2025-02-15 11:41:24,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33363.00 MB 2025-02-15 11:41:24,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6869.09 MB 2025-02-15 11:41:24,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57759.76 MB 2025-02-15 11:41:24,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40120.61 MB 2025-02-15 11:41:24,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17639.15 MB 2025-02-15 11:41:24,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42307.06 MB 2025-02-15 11:41:24,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:41:24,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:41:24,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 11:41:24,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:24,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33363.00 MB 2025-02-15 11:41:24,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25868.50 MB 2025-02-15 11:41:24,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7494.49 MB 2025-02-15 11:41:24,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40120.61 MB 2025-02-15 11:41:24,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54280.59 MB 2025-02-15 11:41:24,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14159.97 MB 2025-02-15 11:41:24,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52527.01 MB 2025-02-15 11:41:26,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:41:26,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:41:26,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:41:26,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:26,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25868.50 MB 2025-02-15 11:41:26,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26399.35 MB 2025-02-15 11:41:26,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:41:26,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54280.59 MB 2025-02-15 11:41:26,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-15 11:41:26,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19614.66 MB 2025-02-15 11:41:26,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30377.89 MB 2025-02-15 11:41:26,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:41:26,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:41:26,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:41:26,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:26,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-15 11:41:26,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28288.88 MB 2025-02-15 11:41:26,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:41:26,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-15 11:41:26,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-15 11:41:26,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:41:26,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.31 MB 2025-02-15 11:41:26,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:41:26,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:41:26,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:41:26,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:26,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28288.88 MB 2025-02-15 11:41:26,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-15 11:41:26,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:41:26,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-15 11:41:26,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39384.51 MB 2025-02-15 11:41:26,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 11:41:26,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-15 11:41:26,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:41:26,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:41:26,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:41:26,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:26,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-15 11:41:26,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-15 11:41:26,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:41:26,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-15 11:41:26,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39384.51 MB 2025-02-15 11:41:26,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 11:41:26,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-15 11:41:27,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:41:27,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:41:27,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 11:41:27,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:27,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32064.28 MB 2025-02-15 11:41:27,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32831.28 MB 2025-02-15 11:41:27,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:41:27,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39384.51 MB 2025-02-15 11:41:27,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39799.75 MB 2025-02-15 11:41:27,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:41:27,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.07 MB 2025-02-15 11:41:27,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:41:27,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:41:27,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:41:27,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:27,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33244.17 MB 2025-02-15 11:41:27,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33473.77 MB 2025-02-15 11:41:27,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.60 MB 2025-02-15 11:41:27,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39799.75 MB 2025-02-15 11:41:27,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39799.75 MB 2025-02-15 11:41:27,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:41:27,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33680.79 MB 2025-02-15 11:41:27,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:41:27,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:41:27,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.56 seconds 2025-02-15 11:41:27,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:27,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19731.31 MB 2025-02-15 11:41:27,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33674.84 MB 2025-02-15 11:41:27,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13943.54 MB 2025-02-15 11:41:27,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57759.76 MB 2025-02-15 11:41:27,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39799.75 MB 2025-02-15 11:41:27,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17960.01 MB 2025-02-15 11:41:27,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33680.79 MB 2025-02-15 11:41:27,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:41:27,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:41:27,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:41:27,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:27,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33674.84 MB 2025-02-15 11:41:27,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24735.69 MB 2025-02-15 11:41:27,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8939.15 MB 2025-02-15 11:41:27,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39799.75 MB 2025-02-15 11:41:27,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39799.75 MB 2025-02-15 11:41:27,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:41:27,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36186.51 MB 2025-02-15 11:41:27,403 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:41:27,403 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:41:27,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:41:27,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:41:27,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:41:27,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:41:27,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24735.69 MB 2025-02-15 11:41:27,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33174.72 MB 2025-02-15 11:41:27,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:41:27,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39799.75 MB 2025-02-15 11:41:27,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48190.46 MB 2025-02-15 11:41:27,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:41:27,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33174.72 MB 2025-02-15 11:41:27,578 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:41:27,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:27,579 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:41:27,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:27,580 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:41:27,585 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:41:27,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:27,586 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:41:27,586 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:41:36,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:36,988 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:41:36,994 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:41:36,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:36,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1593, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:41:36,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:41:36,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1593, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:42:01,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:42:01,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:42:01,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.87 seconds 2025-02-15 11:42:01,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:01,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24068.99 MB 2025-02-15 11:42:01,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.52 MB 2025-02-15 11:42:01,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5637.54 MB 2025-02-15 11:42:01,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60775.46 MB 2025-02-15 11:42:01,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38889.59 MB 2025-02-15 11:42:01,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21885.88 MB 2025-02-15 11:42:01,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38523.19 MB 2025-02-15 11:42:01,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:42:01,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:42:01,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:42:01,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:01,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29706.52 MB 2025-02-15 11:42:01,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.36 MB 2025-02-15 11:42:01,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5647.16 MB 2025-02-15 11:42:01,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38889.59 MB 2025-02-15 11:42:01,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48416.95 MB 2025-02-15 11:42:01,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9527.36 MB 2025-02-15 11:42:01,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44030.08 MB 2025-02-15 11:42:03,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:42:03,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:42:03,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:42:03,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:03,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.36 MB 2025-02-15 11:42:03,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24590.20 MB 2025-02-15 11:42:03,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:42:03,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48416.95 MB 2025-02-15 11:42:03,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33250.34 MB 2025-02-15 11:42:03,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15166.60 MB 2025-02-15 11:42:03,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28568.75 MB 2025-02-15 11:42:03,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:42:03,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:42:03,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:42:03,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:03,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-15 11:42:03,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26479.39 MB 2025-02-15 11:42:03,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.19 MB 2025-02-15 11:42:03,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:42:03,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33250.34 MB 2025-02-15 11:42:03,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:03,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27896.82 MB 2025-02-15 11:42:04,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:42:04,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:42:04,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:42:04,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26479.39 MB 2025-02-15 11:42:04,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.25 MB 2025-02-15 11:42:04,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:42:04,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:42:04,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36081.50 MB 2025-02-15 11:42:04,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:42:04,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.53 MB 2025-02-15 11:42:04,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:42:04,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:42:04,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:42:04,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-15 11:42:04,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.25 MB 2025-02-15 11:42:04,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.04 MB 2025-02-15 11:42:04,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-15 11:42:04,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36081.50 MB 2025-02-15 11:42:04,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:42:04,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.53 MB 2025-02-15 11:42:04,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:42:04,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:42:04,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:42:04,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30254.79 MB 2025-02-15 11:42:04,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31021.79 MB 2025-02-15 11:42:04,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:42:04,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36081.50 MB 2025-02-15 11:42:04,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-15 11:42:04,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:42:04,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31729.58 MB 2025-02-15 11:42:04,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:42:04,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:42:04,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:42:04,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31434.68 MB 2025-02-15 11:42:04,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31662.60 MB 2025-02-15 11:42:04,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-15 11:42:04,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36496.74 MB 2025-02-15 11:42:04,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-15 11:42:04,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:04,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31896.66 MB 2025-02-15 11:42:04,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:42:04,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:42:04,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.31 seconds 2025-02-15 11:42:04,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18518.85 MB 2025-02-15 11:42:04,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31862.56 MB 2025-02-15 11:42:04,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13343.72 MB 2025-02-15 11:42:04,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60775.46 MB 2025-02-15 11:42:04,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-15 11:42:04,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24278.73 MB 2025-02-15 11:42:04,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31896.66 MB 2025-02-15 11:42:04,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:42:04,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:42:04,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:42:04,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31862.56 MB 2025-02-15 11:42:04,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23506.98 MB 2025-02-15 11:42:04,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8355.58 MB 2025-02-15 11:42:04,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36496.74 MB 2025-02-15 11:42:04,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-15 11:42:04,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:04,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34360.41 MB 2025-02-15 11:42:04,604 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-15 11:42:04,604 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:42:04,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:42:04,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:42:04,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:42:04,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:04,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23506.98 MB 2025-02-15 11:42:04,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31899.57 MB 2025-02-15 11:42:04,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-15 11:42:04,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36496.74 MB 2025-02-15 11:42:04,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44841.30 MB 2025-02-15 11:42:04,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-15 11:42:04,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31899.57 MB 2025-02-15 11:42:04,771 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-15 11:42:04,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:04,773 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:42:04,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:04,774 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:42:04,778 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:42:04,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:04,779 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:42:04,780 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:42:14,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:14,417 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:42:14,422 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:42:14,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:14,425 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 167, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:42:14,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:14,426 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 167, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:42:17,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:42:17,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:42:17,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.64 seconds 2025-02-15 11:42:17,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14132.39 MB 2025-02-15 11:42:17,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14723.39 MB 2025-02-15 11:42:17,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 591.00 MB 2025-02-15 11:42:17,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57357.11 MB 2025-02-15 11:42:17,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 11:42:17,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39936.07 MB 2025-02-15 11:42:17,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23603.76 MB 2025-02-15 11:42:17,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:42:17,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:42:17,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:42:17,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14723.39 MB 2025-02-15 11:42:17,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14969.35 MB 2025-02-15 11:42:17,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.96 MB 2025-02-15 11:42:17,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 11:42:17,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 11:42:17,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1132.46 MB 2025-02-15 11:42:17,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16986.63 MB 2025-02-15 11:42:17,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:42:17,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:42:17,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-15 11:42:17,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14969.35 MB 2025-02-15 11:42:17,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15183.01 MB 2025-02-15 11:42:17,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 11:42:17,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 11:42:17,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 11:42:17,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:17,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19140.04 MB 2025-02-15 11:42:17,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:42:17,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:42:17,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 11:42:17,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15182.95 MB 2025-02-15 11:42:17,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15943.30 MB 2025-02-15 11:42:17,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 11:42:17,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 11:42:17,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18553.50 MB 2025-02-15 11:42:17,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:17,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16513.82 MB 2025-02-15 11:42:17,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:42:17,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:42:17,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:42:17,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15943.30 MB 2025-02-15 11:42:17,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.69 MB 2025-02-15 11:42:17,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 11:42:17,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 11:42:17,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20080.23 MB 2025-02-15 11:42:17,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-15 11:42:17,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.22 MB 2025-02-15 11:42:17,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:42:17,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:42:17,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:42:17,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:17,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15182.95 MB 2025-02-15 11:42:17,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.69 MB 2025-02-15 11:42:17,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 11:42:17,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18553.50 MB 2025-02-15 11:42:17,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20080.23 MB 2025-02-15 11:42:17,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-15 11:42:17,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.22 MB 2025-02-15 11:42:18,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:42:18,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:42:18,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:42:18,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:18,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17462.94 MB 2025-02-15 11:42:18,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17771.66 MB 2025-02-15 11:42:18,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 11:42:18,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20080.23 MB 2025-02-15 11:42:18,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20245.91 MB 2025-02-15 11:42:18,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 11:42:18,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18063.88 MB 2025-02-15 11:42:18,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:42:18,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:42:18,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:42:18,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:18,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17937.85 MB 2025-02-15 11:42:18,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18167.04 MB 2025-02-15 11:42:18,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.19 MB 2025-02-15 11:42:18,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20245.91 MB 2025-02-15 11:42:18,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20245.91 MB 2025-02-15 11:42:18,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:18,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18182.22 MB 2025-02-15 11:42:18,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:42:18,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:42:18,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.62 seconds 2025-02-15 11:42:18,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:18,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13550.55 MB 2025-02-15 11:42:18,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18368.11 MB 2025-02-15 11:42:18,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4817.56 MB 2025-02-15 11:42:18,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57357.11 MB 2025-02-15 11:42:18,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20245.91 MB 2025-02-15 11:42:18,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37111.20 MB 2025-02-15 11:42:18,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18368.11 MB 2025-02-15 11:42:18,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:42:18,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:42:18,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:42:18,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:18,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18368.11 MB 2025-02-15 11:42:18,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17427.03 MB 2025-02-15 11:42:18,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -941.09 MB 2025-02-15 11:42:18,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20245.91 MB 2025-02-15 11:42:18,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20245.91 MB 2025-02-15 11:42:18,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:42:18,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19171.85 MB 2025-02-15 11:42:18,333 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:42:18,333 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for the video is 2.'] 2025-02-15 11:42:18,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:42:18,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:42:18,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:42:18,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:42:18,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17427.03 MB 2025-02-15 11:42:18,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25866.05 MB 2025-02-15 11:42:18,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:42:18,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20245.91 MB 2025-02-15 11:42:18,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30735.86 MB 2025-02-15 11:42:18,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:42:18,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25866.05 MB 2025-02-15 11:42:18,501 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:42:18,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:18,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:42:18,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:18,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:42:18,508 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:42:18,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:42:18,509 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:42:18,510 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for the video is 2.'] 2025-02-15 11:43:35,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:35,895 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:43:35,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:43:35,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:35,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:43:35,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:35,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:43:38,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:43:38,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:43:38,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.76 seconds 2025-02-15 11:43:38,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:38,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-15 11:43:38,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-15 11:43:38,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-15 11:43:38,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43320.87 MB 2025-02-15 11:43:38,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19641.93 MB 2025-02-15 11:43:38,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23678.94 MB 2025-02-15 11:43:38,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-15 11:43:38,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:43:38,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:43:38,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:43:38,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:38,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-15 11:43:38,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15142.35 MB 2025-02-15 11:43:38,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.87 MB 2025-02-15 11:43:38,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19641.93 MB 2025-02-15 11:43:38,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19641.93 MB 2025-02-15 11:43:38,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:43:38,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17335.69 MB 2025-02-15 11:43:39,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:43:39,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:43:39,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 11:43:39,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15142.35 MB 2025-02-15 11:43:39,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.25 MB 2025-02-15 11:43:39,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 11:43:39,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19641.93 MB 2025-02-15 11:43:39,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-15 11:43:39,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -381.68 MB 2025-02-15 11:43:39,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.04 MB 2025-02-15 11:43:39,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:43:39,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:43:39,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:43:39,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-15 11:43:39,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.10 MB 2025-02-15 11:43:39,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 11:43:39,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-15 11:43:39,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-15 11:43:39,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:43:39,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.31 MB 2025-02-15 11:43:39,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:43:39,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:43:39,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:43:39,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.10 MB 2025-02-15 11:43:39,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-15 11:43:39,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 11:43:39,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-15 11:43:39,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 11:43:39,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 11:43:39,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.38 MB 2025-02-15 11:43:39,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:43:39,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:43:39,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:43:39,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-15 11:43:39,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-15 11:43:39,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 11:43:39,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-15 11:43:39,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 11:43:39,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 11:43:39,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.38 MB 2025-02-15 11:43:39,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:43:39,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:43:39,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 11:43:39,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.75 MB 2025-02-15 11:43:39,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.06 MB 2025-02-15 11:43:39,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 11:43:39,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 11:43:39,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21330.13 MB 2025-02-15 11:43:39,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 11:43:39,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18543.27 MB 2025-02-15 11:43:39,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:43:39,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:43:39,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:43:39,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18406.77 MB 2025-02-15 11:43:39,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18633.92 MB 2025-02-15 11:43:39,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.14 MB 2025-02-15 11:43:39,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21330.13 MB 2025-02-15 11:43:39,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21330.13 MB 2025-02-15 11:43:39,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:43:39,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18665.82 MB 2025-02-15 11:43:39,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:43:39,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:43:39,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.81 seconds 2025-02-15 11:43:39,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-15 11:43:39,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.99 MB 2025-02-15 11:43:39,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.63 MB 2025-02-15 11:43:39,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43320.87 MB 2025-02-15 11:43:39,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21332.23 MB 2025-02-15 11:43:39,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21988.64 MB 2025-02-15 11:43:39,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18834.99 MB 2025-02-15 11:43:39,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:43:39,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:43:39,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:43:39,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:39,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18834.99 MB 2025-02-15 11:43:39,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17545.26 MB 2025-02-15 11:43:39,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1289.73 MB 2025-02-15 11:43:39,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21332.23 MB 2025-02-15 11:43:39,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21332.23 MB 2025-02-15 11:43:39,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:43:39,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19070.97 MB 2025-02-15 11:43:40,008 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:43:40,009 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 11:43:40,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:43:40,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:43:40,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:43:40,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:43:40,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17545.26 MB 2025-02-15 11:43:40,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25984.28 MB 2025-02-15 11:43:40,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:43:40,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21332.23 MB 2025-02-15 11:43:40,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31822.18 MB 2025-02-15 11:43:40,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:43:40,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25984.28 MB 2025-02-15 11:43:40,177 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:43:40,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:40,178 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:43:40,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:40,179 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:43:40,184 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:43:40,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:40,185 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:43:40,185 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 11:43:57,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:57,059 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:43:57,064 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:43:57,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:57,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1762, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:43:57,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:43:57,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1762, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:44:24,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:44:24,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:44:24,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.28 seconds 2025-02-15 11:44:24,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:24,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25246.60 MB 2025-02-15 11:44:24,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31482.22 MB 2025-02-15 11:44:24,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6235.62 MB 2025-02-15 11:44:24,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44407.19 MB 2025-02-15 11:44:24,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39493.57 MB 2025-02-15 11:44:24,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4913.63 MB 2025-02-15 11:44:24,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40380.29 MB 2025-02-15 11:44:24,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:44:24,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:44:24,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 11:44:24,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:24,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31482.22 MB 2025-02-15 11:44:24,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24937.94 MB 2025-02-15 11:44:24,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6544.28 MB 2025-02-15 11:44:24,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39493.57 MB 2025-02-15 11:44:24,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52451.87 MB 2025-02-15 11:44:24,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12958.30 MB 2025-02-15 11:44:24,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48838.72 MB 2025-02-15 11:44:26,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:44:26,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:44:26,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:44:26,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.94 MB 2025-02-15 11:44:26,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25468.78 MB 2025-02-15 11:44:26,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:44:26,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52451.87 MB 2025-02-15 11:44:26,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:44:26,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17779.65 MB 2025-02-15 11:44:26,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.33 MB 2025-02-15 11:44:26,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:44:26,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:44:26,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:44:26,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-15 11:44:26,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27358.31 MB 2025-02-15 11:44:26,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:44:26,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:44:26,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-15 11:44:26,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:44:26,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28775.74 MB 2025-02-15 11:44:26,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:44:26,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:44:26,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:44:26,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27358.31 MB 2025-02-15 11:44:26,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-15 11:44:26,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:44:26,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:44:26,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37503.37 MB 2025-02-15 11:44:26,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:44:26,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-15 11:44:26,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:44:26,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:44:26,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:44:26,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-15 11:44:26,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-15 11:44:26,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:44:26,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-15 11:44:26,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37503.37 MB 2025-02-15 11:44:26,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:44:26,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-15 11:44:26,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:44:26,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:44:26,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:44:26,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31133.71 MB 2025-02-15 11:44:26,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31900.71 MB 2025-02-15 11:44:26,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:44:26,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37503.37 MB 2025-02-15 11:44:26,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 11:44:26,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:44:26,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32608.50 MB 2025-02-15 11:44:26,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:44:26,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:44:26,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:44:26,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32313.60 MB 2025-02-15 11:44:26,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32542.29 MB 2025-02-15 11:44:26,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-15 11:44:26,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 11:44:26,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 11:44:26,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:44:26,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32763.19 MB 2025-02-15 11:44:26,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:44:26,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:44:26,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.76 seconds 2025-02-15 11:44:26,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:26,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19107.66 MB 2025-02-15 11:44:26,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32742.90 MB 2025-02-15 11:44:26,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13635.24 MB 2025-02-15 11:44:26,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44407.19 MB 2025-02-15 11:44:26,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 11:44:26,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6486.49 MB 2025-02-15 11:44:26,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32763.19 MB 2025-02-15 11:44:27,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:44:27,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:44:27,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:44:27,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:27,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32742.90 MB 2025-02-15 11:44:27,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24105.05 MB 2025-02-15 11:44:27,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8637.85 MB 2025-02-15 11:44:27,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 11:44:27,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 11:44:27,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:44:27,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35248.98 MB 2025-02-15 11:44:27,124 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 11:44:27,125 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:44:27,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:44:27,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:44:27,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:44:27,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:44:27,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24105.05 MB 2025-02-15 11:44:27,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32524.13 MB 2025-02-15 11:44:27,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-15 11:44:27,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 11:44:27,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46292.53 MB 2025-02-15 11:44:27,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 11:44:27,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32524.13 MB 2025-02-15 11:44:27,291 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 11:44:27,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:44:27,293 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:44:27,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:44:27,294 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:44:27,298 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:44:27,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:44:27,299 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:44:27,300 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:46:12,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:12,883 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:46:12,888 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:46:12,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:12,892 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 352, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:46:12,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:12,893 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 352, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:46:18,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:46:18,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:46:18,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.42 seconds 2025-02-15 11:46:18,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:18,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.50 MB 2025-02-15 11:46:18,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16667.21 MB 2025-02-15 11:46:18,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1245.71 MB 2025-02-15 11:46:18,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54664.36 MB 2025-02-15 11:46:18,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21420.31 MB 2025-02-15 11:46:18,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33244.05 MB 2025-02-15 11:46:18,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25572.35 MB 2025-02-15 11:46:18,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:46:18,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:46:18,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 11:46:18,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:18,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16667.21 MB 2025-02-15 11:46:18,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17270.68 MB 2025-02-15 11:46:18,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 603.48 MB 2025-02-15 11:46:18,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21420.31 MB 2025-02-15 11:46:18,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23911.73 MB 2025-02-15 11:46:18,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2491.42 MB 2025-02-15 11:46:18,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21611.41 MB 2025-02-15 11:46:20,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:46:20,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:46:20,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-15 11:46:20,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17270.68 MB 2025-02-15 11:46:20,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17737.82 MB 2025-02-15 11:46:20,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 467.14 MB 2025-02-15 11:46:20,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23911.73 MB 2025-02-15 11:46:20,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20084.42 MB 2025-02-15 11:46:20,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3827.30 MB 2025-02-15 11:46:20,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21696.18 MB 2025-02-15 11:46:20,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:46:20,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:46:20,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:46:20,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17737.82 MB 2025-02-15 11:46:20,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19400.87 MB 2025-02-15 11:46:20,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1663.04 MB 2025-02-15 11:46:20,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20084.42 MB 2025-02-15 11:46:20,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22575.84 MB 2025-02-15 11:46:20,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2491.42 MB 2025-02-15 11:46:20,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20648.20 MB 2025-02-15 11:46:20,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:46:20,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:46:20,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 11:46:20,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19400.87 MB 2025-02-15 11:46:20,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.58 MB 2025-02-15 11:46:20,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1973.72 MB 2025-02-15 11:46:20,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22575.84 MB 2025-02-15 11:46:20,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28181.53 MB 2025-02-15 11:46:20,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5605.69 MB 2025-02-15 11:46:20,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.54 MB 2025-02-15 11:46:20,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:46:20,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:46:20,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 11:46:20,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17737.82 MB 2025-02-15 11:46:20,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.58 MB 2025-02-15 11:46:20,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3636.76 MB 2025-02-15 11:46:20,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20084.42 MB 2025-02-15 11:46:20,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28181.53 MB 2025-02-15 11:46:20,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8097.10 MB 2025-02-15 11:46:20,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.54 MB 2025-02-15 11:46:20,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:46:20,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:46:20,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 11:46:20,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22724.10 MB 2025-02-15 11:46:20,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23399.06 MB 2025-02-15 11:46:20,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 674.96 MB 2025-02-15 11:46:20,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28181.53 MB 2025-02-15 11:46:20,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28546.43 MB 2025-02-15 11:46:20,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 364.90 MB 2025-02-15 11:46:20,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24021.91 MB 2025-02-15 11:46:20,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:46:20,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:46:20,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:46:20,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23762.40 MB 2025-02-15 11:46:20,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23988.90 MB 2025-02-15 11:46:20,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.50 MB 2025-02-15 11:46:20,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28546.43 MB 2025-02-15 11:46:20,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28546.43 MB 2025-02-15 11:46:20,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:46:20,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24158.77 MB 2025-02-15 11:46:20,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:46:20,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:46:20,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.53 seconds 2025-02-15 11:46:20,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-15 11:46:20,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24189.97 MB 2025-02-15 11:46:20,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9994.87 MB 2025-02-15 11:46:20,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54664.36 MB 2025-02-15 11:46:20,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28546.43 MB 2025-02-15 11:46:20,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26117.93 MB 2025-02-15 11:46:20,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24189.97 MB 2025-02-15 11:46:20,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:46:20,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:46:20,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:46:20,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24189.97 MB 2025-02-15 11:46:20,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18972.97 MB 2025-02-15 11:46:20,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5217.01 MB 2025-02-15 11:46:20,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28546.43 MB 2025-02-15 11:46:20,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28546.43 MB 2025-02-15 11:46:20,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:46:20,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27404.91 MB 2025-02-15 11:46:20,708 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:46:20,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:46:20,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:46:20,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:46:20,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:46:20,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:46:20,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18972.97 MB 2025-02-15 11:46:20,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27411.99 MB 2025-02-15 11:46:20,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:46:20,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28546.43 MB 2025-02-15 11:46:20,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36937.14 MB 2025-02-15 11:46:20,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 11:46:20,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27411.99 MB 2025-02-15 11:46:20,876 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:46:20,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:20,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:46:20,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:20,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:46:20,883 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:46:20,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:46:20,884 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:46:20,884 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:47:57,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:47:57,525 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:47:57,531 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:47:57,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:47:57,535 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2619, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:47:57,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:47:57,536 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2619, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:48:37,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:48:37,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:48:37,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.33 seconds 2025-02-15 11:48:37,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:37,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31219.23 MB 2025-02-15 11:48:37,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40488.65 MB 2025-02-15 11:48:37,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9269.41 MB 2025-02-15 11:48:37,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67775.76 MB 2025-02-15 11:48:37,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44002.44 MB 2025-02-15 11:48:37,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23773.32 MB 2025-02-15 11:48:37,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49757.14 MB 2025-02-15 11:48:38,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:48:38,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:48:38,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 11:48:38,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:38,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40488.65 MB 2025-02-15 11:48:38,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29394.72 MB 2025-02-15 11:48:38,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11093.92 MB 2025-02-15 11:48:38,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44002.44 MB 2025-02-15 11:48:38,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63954.75 MB 2025-02-15 11:48:38,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19952.30 MB 2025-02-15 11:48:38,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67301.78 MB 2025-02-15 11:48:40,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:48:40,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:48:40,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 11:48:40,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29394.72 MB 2025-02-15 11:48:40,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29925.56 MB 2025-02-15 11:48:40,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:48:40,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63954.75 MB 2025-02-15 11:48:40,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32222.74 MB 2025-02-15 11:48:40,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31732.01 MB 2025-02-15 11:48:40,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33904.11 MB 2025-02-15 11:48:40,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:48:40,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:48:40,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:48:40,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29925.56 MB 2025-02-15 11:48:40,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31815.10 MB 2025-02-15 11:48:40,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:48:40,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32222.74 MB 2025-02-15 11:48:40,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35053.90 MB 2025-02-15 11:48:40,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:48:40,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33232.53 MB 2025-02-15 11:48:40,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:48:40,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:48:40,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:48:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31815.10 MB 2025-02-15 11:48:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34056.95 MB 2025-02-15 11:48:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:48:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35053.90 MB 2025-02-15 11:48:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41188.07 MB 2025-02-15 11:48:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:48:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39601.23 MB 2025-02-15 11:48:40,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:48:40,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:48:40,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:48:40,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29925.56 MB 2025-02-15 11:48:40,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34056.95 MB 2025-02-15 11:48:40,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:48:40,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32222.74 MB 2025-02-15 11:48:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41188.07 MB 2025-02-15 11:48:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 11:48:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39601.23 MB 2025-02-15 11:48:40,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:48:40,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:48:40,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 11:48:40,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35590.50 MB 2025-02-15 11:48:40,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36357.50 MB 2025-02-15 11:48:40,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:48:40,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41188.07 MB 2025-02-15 11:48:40,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41603.30 MB 2025-02-15 11:48:40,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:48:40,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37065.29 MB 2025-02-15 11:48:40,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:48:40,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:48:40,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 11:48:40,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36770.39 MB 2025-02-15 11:48:40,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36998.96 MB 2025-02-15 11:48:40,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.58 MB 2025-02-15 11:48:40,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41603.30 MB 2025-02-15 11:48:40,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41603.30 MB 2025-02-15 11:48:40,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:48:40,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37211.76 MB 2025-02-15 11:48:40,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:48:40,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:48:40,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.11 seconds 2025-02-15 11:48:40,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22093.97 MB 2025-02-15 11:48:40,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37200.01 MB 2025-02-15 11:48:40,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15106.04 MB 2025-02-15 11:48:40,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58648.95 MB 2025-02-15 11:48:40,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41603.30 MB 2025-02-15 11:48:40,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17045.65 MB 2025-02-15 11:48:40,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37211.76 MB 2025-02-15 11:48:40,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:48:40,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:48:40,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 11:48:40,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37200.01 MB 2025-02-15 11:48:40,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27097.42 MB 2025-02-15 11:48:40,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10102.59 MB 2025-02-15 11:48:40,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41603.30 MB 2025-02-15 11:48:40,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41603.30 MB 2025-02-15 11:48:40,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:48:40,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39711.37 MB 2025-02-15 11:48:40,962 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 11:48:40,963 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:48:40,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:48:40,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:48:40,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:48:40,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:48:40,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27097.42 MB 2025-02-15 11:48:40,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35535.41 MB 2025-02-15 11:48:40,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.99 MB 2025-02-15 11:48:40,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41603.30 MB 2025-02-15 11:48:40,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45797.61 MB 2025-02-15 11:48:40,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 11:48:40,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35535.41 MB 2025-02-15 11:48:41,190 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 11:48:41,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:41,192 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:48:41,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:41,193 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:48:41,198 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:48:41,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:41,200 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:48:41,200 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:48:51,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:51,582 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:48:51,587 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:48:51,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:51,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2568, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:48:51,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:48:51,592 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2568, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:49:31,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:49:31,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:49:31,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.18 seconds 2025-02-15 11:49:31,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:31,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30864.37 MB 2025-02-15 11:49:31,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39953.43 MB 2025-02-15 11:49:31,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9089.06 MB 2025-02-15 11:49:31,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63134.76 MB 2025-02-15 11:49:31,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43645.93 MB 2025-02-15 11:49:31,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19488.83 MB 2025-02-15 11:49:31,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49041.44 MB 2025-02-15 11:49:32,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:49:32,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:49:32,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:49:32,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:32,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39953.43 MB 2025-02-15 11:49:32,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29130.56 MB 2025-02-15 11:49:32,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10822.87 MB 2025-02-15 11:49:32,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43645.93 MB 2025-02-15 11:49:32,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62142.81 MB 2025-02-15 11:49:32,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18496.88 MB 2025-02-15 11:49:32,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64582.38 MB 2025-02-15 11:49:33,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:49:33,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:49:33,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:49:33,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:33,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29130.56 MB 2025-02-15 11:49:33,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29661.40 MB 2025-02-15 11:49:33,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:49:33,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62142.81 MB 2025-02-15 11:49:33,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32046.58 MB 2025-02-15 11:49:33,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30096.23 MB 2025-02-15 11:49:33,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33639.95 MB 2025-02-15 11:49:34,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:49:34,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:49:34,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:49:34,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29661.40 MB 2025-02-15 11:49:34,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31550.94 MB 2025-02-15 11:49:34,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:49:34,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32046.58 MB 2025-02-15 11:49:34,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-15 11:49:34,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:49:34,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32968.36 MB 2025-02-15 11:49:34,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:49:34,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:49:34,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:49:34,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31550.94 MB 2025-02-15 11:49:34,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33792.79 MB 2025-02-15 11:49:34,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:49:34,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-15 11:49:34,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41011.90 MB 2025-02-15 11:49:34,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:49:34,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39337.07 MB 2025-02-15 11:49:34,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:49:34,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:49:34,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 11:49:34,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29661.40 MB 2025-02-15 11:49:34,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33792.79 MB 2025-02-15 11:49:34,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:49:34,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32046.58 MB 2025-02-15 11:49:34,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41011.90 MB 2025-02-15 11:49:34,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 11:49:34,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39337.07 MB 2025-02-15 11:49:34,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:49:34,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:49:34,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:49:34,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35326.33 MB 2025-02-15 11:49:34,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36093.34 MB 2025-02-15 11:49:34,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:49:34,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41011.90 MB 2025-02-15 11:49:34,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41425.04 MB 2025-02-15 11:49:34,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 11:49:34,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36801.12 MB 2025-02-15 11:49:34,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:49:34,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:49:34,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:49:34,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36506.22 MB 2025-02-15 11:49:34,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36734.46 MB 2025-02-15 11:49:34,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.23 MB 2025-02-15 11:49:34,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41425.04 MB 2025-02-15 11:49:34,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41425.04 MB 2025-02-15 11:49:34,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:49:34,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36945.69 MB 2025-02-15 11:49:34,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:49:34,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:49:34,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.82 seconds 2025-02-15 11:49:34,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21917.25 MB 2025-02-15 11:49:34,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36934.45 MB 2025-02-15 11:49:34,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15017.19 MB 2025-02-15 11:49:34,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63134.76 MB 2025-02-15 11:49:34,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41425.04 MB 2025-02-15 11:49:34,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21709.72 MB 2025-02-15 11:49:34,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36945.69 MB 2025-02-15 11:49:34,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:49:34,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:49:34,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:49:34,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36934.45 MB 2025-02-15 11:49:34,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26905.74 MB 2025-02-15 11:49:34,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10028.71 MB 2025-02-15 11:49:34,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41425.04 MB 2025-02-15 11:49:34,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41425.04 MB 2025-02-15 11:49:34,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:49:34,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39432.60 MB 2025-02-15 11:49:34,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-15 11:49:34,707 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:49:34,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:49:34,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:49:34,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:49:34,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:49:34,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26905.74 MB 2025-02-15 11:49:34,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35299.02 MB 2025-02-15 11:49:34,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-15 11:49:34,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41425.04 MB 2025-02-15 11:49:34,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45598.38 MB 2025-02-15 11:49:34,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 11:49:34,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35299.02 MB 2025-02-15 11:49:34,873 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-15 11:49:34,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:49:34,874 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:49:34,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:49:34,875 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:49:34,881 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:49:34,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:49:34,882 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:49:34,882 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:51:07,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:07,521 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:51:07,530 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:51:07,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:07,537 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:51:07,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:07,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:51:10,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:51:10,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:51:10,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.95 seconds 2025-02-15 11:51:10,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:10,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-15 11:51:10,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-15 11:51:10,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-15 11:51:10,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53945.04 MB 2025-02-15 11:51:10,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19144.90 MB 2025-02-15 11:51:10,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34800.14 MB 2025-02-15 11:51:10,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-15 11:51:10,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:51:10,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:51:10,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:51:10,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:10,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-15 11:51:10,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15241.94 MB 2025-02-15 11:51:10,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.92 MB 2025-02-15 11:51:10,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19144.90 MB 2025-02-15 11:51:10,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19144.90 MB 2025-02-15 11:51:10,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:51:10,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17535.66 MB 2025-02-15 11:51:11,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:51:11,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:51:11,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.92 seconds 2025-02-15 11:51:11,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.94 MB 2025-02-15 11:51:11,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15488.79 MB 2025-02-15 11:51:11,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-15 11:51:11,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19144.90 MB 2025-02-15 11:51:11,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18025.02 MB 2025-02-15 11:51:11,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1119.88 MB 2025-02-15 11:51:11,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19412.63 MB 2025-02-15 11:51:11,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:51:11,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:51:11,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:51:11,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-15 11:51:11,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.14 MB 2025-02-15 11:51:11,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-15 11:51:11,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18025.02 MB 2025-02-15 11:51:11,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18905.83 MB 2025-02-15 11:51:11,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 880.80 MB 2025-02-15 11:51:11,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.25 MB 2025-02-15 11:51:11,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:51:11,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:51:11,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:51:11,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.14 MB 2025-02-15 11:51:11,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-15 11:51:11,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-15 11:51:11,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18905.83 MB 2025-02-15 11:51:11,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 11:51:11,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2642.41 MB 2025-02-15 11:51:11,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19989.27 MB 2025-02-15 11:51:11,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:51:11,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:51:11,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 11:51:11,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-15 11:51:11,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-15 11:51:11,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-15 11:51:11,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18025.02 MB 2025-02-15 11:51:11,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-15 11:51:11,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3523.22 MB 2025-02-15 11:51:11,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19989.27 MB 2025-02-15 11:51:11,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:51:11,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:51:11,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:51:11,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18122.74 MB 2025-02-15 11:51:11,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18480.96 MB 2025-02-15 11:51:11,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 358.23 MB 2025-02-15 11:51:11,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21548.24 MB 2025-02-15 11:51:11,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21739.08 MB 2025-02-15 11:51:11,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-15 11:51:11,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18815.71 MB 2025-02-15 11:51:11,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:51:11,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:51:11,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:51:11,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18672.96 MB 2025-02-15 11:51:11,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18876.62 MB 2025-02-15 11:51:11,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.65 MB 2025-02-15 11:51:11,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21739.08 MB 2025-02-15 11:51:11,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21743.27 MB 2025-02-15 11:51:11,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 11:51:11,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18904.36 MB 2025-02-15 11:51:11,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:51:11,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:51:11,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.12 seconds 2025-02-15 11:51:11,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-15 11:51:11,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19077.52 MB 2025-02-15 11:51:11,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5460.77 MB 2025-02-15 11:51:11,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53945.04 MB 2025-02-15 11:51:11,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21743.27 MB 2025-02-15 11:51:11,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32201.77 MB 2025-02-15 11:51:11,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.52 MB 2025-02-15 11:51:11,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:51:11,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:51:11,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 11:51:11,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.52 MB 2025-02-15 11:51:11,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17609.59 MB 2025-02-15 11:51:11,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1467.93 MB 2025-02-15 11:51:11,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21743.27 MB 2025-02-15 11:51:11,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21743.27 MB 2025-02-15 11:51:11,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:51:11,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.52 MB 2025-02-15 11:51:11,943 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 11:51:11,943 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:51:11,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:51:11,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:51:11,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:51:11,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:51:11,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.59 MB 2025-02-15 11:51:11,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26041.05 MB 2025-02-15 11:51:11,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 11:51:11,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21743.27 MB 2025-02-15 11:51:11,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32224.84 MB 2025-02-15 11:51:11,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 11:51:11,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26041.05 MB 2025-02-15 11:51:12,113 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 11:51:12,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:12,114 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:51:12,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:12,115 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:51:12,120 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:51:12,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:51:12,121 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:51:12,121 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:52:06,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:06,290 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:52:06,295 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:52:06,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:06,299 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:52:06,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:06,300 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:52:37,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:52:37,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:52:37,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.90 seconds 2025-02-15 11:52:37,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:37,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-15 11:52:37,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-15 11:52:37,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-15 11:52:37,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40609.25 MB 2025-02-15 11:52:37,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40340.82 MB 2025-02-15 11:52:37,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -268.44 MB 2025-02-15 11:52:37,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42972.55 MB 2025-02-15 11:52:37,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:52:37,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:52:37,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:52:37,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:37,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-15 11:52:37,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26196.02 MB 2025-02-15 11:52:37,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7829.45 MB 2025-02-15 11:52:37,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40340.82 MB 2025-02-15 11:52:37,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55404.66 MB 2025-02-15 11:52:37,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15063.84 MB 2025-02-15 11:52:37,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54548.72 MB 2025-02-15 11:52:39,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:52:39,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:52:39,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 11:52:39,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26196.02 MB 2025-02-15 11:52:39,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.86 MB 2025-02-15 11:52:39,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:52:39,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55404.66 MB 2025-02-15 11:52:39,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30471.62 MB 2025-02-15 11:52:39,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24933.04 MB 2025-02-15 11:52:39,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30706.45 MB 2025-02-15 11:52:39,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:52:39,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:52:39,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:52:39,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26726.86 MB 2025-02-15 11:52:39,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.40 MB 2025-02-15 11:52:39,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:52:39,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30471.62 MB 2025-02-15 11:52:39,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32359.06 MB 2025-02-15 11:52:39,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 11:52:39,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30033.83 MB 2025-02-15 11:52:39,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:52:39,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:52:39,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:52:39,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28616.40 MB 2025-02-15 11:52:39,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30858.25 MB 2025-02-15 11:52:39,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:52:39,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32359.06 MB 2025-02-15 11:52:39,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38493.22 MB 2025-02-15 11:52:39,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 11:52:39,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36402.53 MB 2025-02-15 11:52:39,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:52:39,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:52:39,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:52:39,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26726.86 MB 2025-02-15 11:52:39,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30858.25 MB 2025-02-15 11:52:39,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:52:39,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30471.62 MB 2025-02-15 11:52:39,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38493.22 MB 2025-02-15 11:52:39,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 11:52:39,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36402.53 MB 2025-02-15 11:52:39,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:52:39,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:52:39,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:52:39,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32391.80 MB 2025-02-15 11:52:39,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33158.80 MB 2025-02-15 11:52:39,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:52:39,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38493.22 MB 2025-02-15 11:52:39,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-15 11:52:39,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:52:39,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33866.59 MB 2025-02-15 11:52:39,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:52:39,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:52:39,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:52:39,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33571.69 MB 2025-02-15 11:52:39,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33799.64 MB 2025-02-15 11:52:39,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-15 11:52:39,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-15 11:52:39,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-15 11:52:39,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:52:39,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34008.43 MB 2025-02-15 11:52:39,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:52:39,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:52:39,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.43 seconds 2025-02-15 11:52:39,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:39,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-15 11:52:39,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34000.49 MB 2025-02-15 11:52:39,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14049.69 MB 2025-02-15 11:52:39,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40609.25 MB 2025-02-15 11:52:39,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-15 11:52:39,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1700.79 MB 2025-02-15 11:52:39,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34008.43 MB 2025-02-15 11:52:40,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:52:40,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:52:40,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:52:40,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:40,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34000.49 MB 2025-02-15 11:52:40,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24937.51 MB 2025-02-15 11:52:40,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9062.98 MB 2025-02-15 11:52:40,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-15 11:52:40,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-15 11:52:40,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:52:40,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36497.10 MB 2025-02-15 11:52:40,024 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 11:52:40,024 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:52:40,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:52:40,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:52:40,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:52:40,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:52:40,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.51 MB 2025-02-15 11:52:40,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33325.62 MB 2025-02-15 11:52:40,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.11 MB 2025-02-15 11:52:40,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-15 11:52:40,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43079.70 MB 2025-02-15 11:52:40,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 11:52:40,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33325.62 MB 2025-02-15 11:52:40,188 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 11:52:40,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:40,190 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:52:40,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:40,191 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:52:40,196 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:52:40,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:52:40,197 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:52:40,197 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:53:54,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:53:54,221 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:53:54,227 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:53:54,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:53:54,232 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1369, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:53:54,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:53:54,234 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1369, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:54:15,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:54:15,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:54:15,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.12 seconds 2025-02-15 11:54:15,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:15,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22508.12 MB 2025-02-15 11:54:15,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27352.93 MB 2025-02-15 11:54:15,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4844.81 MB 2025-02-15 11:54:15,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51417.97 MB 2025-02-15 11:54:15,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-15 11:54:15,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13390.32 MB 2025-02-15 11:54:15,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36282.84 MB 2025-02-15 11:54:15,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:54:15,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:54:15,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:54:15,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:15,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27352.93 MB 2025-02-15 11:54:15,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22894.86 MB 2025-02-15 11:54:15,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4458.08 MB 2025-02-15 11:54:15,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-15 11:54:15,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47372.57 MB 2025-02-15 11:54:15,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9344.91 MB 2025-02-15 11:54:15,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41401.14 MB 2025-02-15 11:54:17,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:54:17,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:54:17,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:54:17,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22894.86 MB 2025-02-15 11:54:17,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23425.70 MB 2025-02-15 11:54:17,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:54:17,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47372.57 MB 2025-02-15 11:54:17,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29009.90 MB 2025-02-15 11:54:17,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18362.66 MB 2025-02-15 11:54:17,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27404.24 MB 2025-02-15 11:54:17,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:54:17,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:54:17,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:54:17,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23425.70 MB 2025-02-15 11:54:17,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25315.23 MB 2025-02-15 11:54:17,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:54:17,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29009.90 MB 2025-02-15 11:54:17,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29953.62 MB 2025-02-15 11:54:17,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 11:54:17,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26732.66 MB 2025-02-15 11:54:17,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:54:17,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:54:17,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:54:17,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25315.23 MB 2025-02-15 11:54:17,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27557.09 MB 2025-02-15 11:54:17,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:54:17,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29953.62 MB 2025-02-15 11:54:17,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35615.93 MB 2025-02-15 11:54:17,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 11:54:17,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33101.37 MB 2025-02-15 11:54:17,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:54:17,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:54:17,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 11:54:17,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23425.70 MB 2025-02-15 11:54:17,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27557.09 MB 2025-02-15 11:54:17,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:54:17,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29009.90 MB 2025-02-15 11:54:17,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35615.93 MB 2025-02-15 11:54:17,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 11:54:17,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33101.37 MB 2025-02-15 11:54:17,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:54:17,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:54:17,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 11:54:17,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29090.63 MB 2025-02-15 11:54:17,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29857.63 MB 2025-02-15 11:54:17,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:54:17,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35615.93 MB 2025-02-15 11:54:17,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36031.17 MB 2025-02-15 11:54:17,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:54:17,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30565.42 MB 2025-02-15 11:54:17,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:54:17,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:54:17,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:54:17,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30270.52 MB 2025-02-15 11:54:17,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30499.60 MB 2025-02-15 11:54:17,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-15 11:54:17,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36031.17 MB 2025-02-15 11:54:17,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36031.17 MB 2025-02-15 11:54:17,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:54:17,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30730.71 MB 2025-02-15 11:54:17,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:54:17,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:54:17,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.56 seconds 2025-02-15 11:54:17,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:17,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.41 MB 2025-02-15 11:54:17,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30700.60 MB 2025-02-15 11:54:17,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12962.19 MB 2025-02-15 11:54:17,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51417.97 MB 2025-02-15 11:54:17,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36031.17 MB 2025-02-15 11:54:17,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15386.80 MB 2025-02-15 11:54:17,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30730.71 MB 2025-02-15 11:54:18,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:54:18,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:54:18,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:54:18,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:18,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30700.60 MB 2025-02-15 11:54:18,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22741.66 MB 2025-02-15 11:54:18,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7958.94 MB 2025-02-15 11:54:18,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36031.17 MB 2025-02-15 11:54:18,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36031.17 MB 2025-02-15 11:54:18,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:54:18,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33211.35 MB 2025-02-15 11:54:18,082 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 11:54:18,083 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:54:18,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:54:18,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:54:18,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:54:18,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:18,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22741.66 MB 2025-02-15 11:54:18,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31177.25 MB 2025-02-15 11:54:18,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 11:54:18,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36031.17 MB 2025-02-15 11:54:18,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44419.78 MB 2025-02-15 11:54:18,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 11:54:18,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31177.25 MB 2025-02-15 11:54:18,246 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 11:54:18,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:18,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:54:18,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:18,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:54:18,253 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:54:18,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:18,255 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:54:18,255 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:54:28,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:28,140 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:54:28,148 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:54:28,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:28,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1537, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:54:28,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:28,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1537, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:54:52,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:54:52,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:54:52,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.06 seconds 2025-02-15 11:54:52,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:52,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23678.77 MB 2025-02-15 11:54:52,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29118.78 MB 2025-02-15 11:54:52,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5440.01 MB 2025-02-15 11:54:52,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52808.38 MB 2025-02-15 11:54:52,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38644.22 MB 2025-02-15 11:54:52,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14164.16 MB 2025-02-15 11:54:52,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38132.97 MB 2025-02-15 11:54:52,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:54:52,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:54:52,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 11:54:52,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:52,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29118.78 MB 2025-02-15 11:54:52,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23768.23 MB 2025-02-15 11:54:52,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5350.55 MB 2025-02-15 11:54:52,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38644.22 MB 2025-02-15 11:54:52,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49081.75 MB 2025-02-15 11:54:52,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10437.53 MB 2025-02-15 11:54:52,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44815.44 MB 2025-02-15 11:54:54,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:54:54,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:54:54,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:54:54,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23768.23 MB 2025-02-15 11:54:54,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24299.08 MB 2025-02-15 11:54:54,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:54:54,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49081.75 MB 2025-02-15 11:54:54,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33204.21 MB 2025-02-15 11:54:54,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15877.54 MB 2025-02-15 11:54:54,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28277.62 MB 2025-02-15 11:54:54,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:54:54,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:54:54,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:54:54,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24299.08 MB 2025-02-15 11:54:54,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26188.61 MB 2025-02-15 11:54:54,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:54:54,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33204.21 MB 2025-02-15 11:54:54,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33204.21 MB 2025-02-15 11:54:54,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:54:54,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27606.04 MB 2025-02-15 11:54:54,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:54:54,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:54:54,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:54:54,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26188.61 MB 2025-02-15 11:54:54,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28430.71 MB 2025-02-15 11:54:54,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.10 MB 2025-02-15 11:54:54,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33204.21 MB 2025-02-15 11:54:54,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36035.36 MB 2025-02-15 11:54:54,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:54:54,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33974.99 MB 2025-02-15 11:54:54,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:54:54,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:54:54,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:54:54,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24299.08 MB 2025-02-15 11:54:54,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28430.71 MB 2025-02-15 11:54:54,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.63 MB 2025-02-15 11:54:54,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33204.21 MB 2025-02-15 11:54:54,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36035.36 MB 2025-02-15 11:54:54,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 11:54:54,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33974.99 MB 2025-02-15 11:54:54,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:54:54,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:54:54,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:54:54,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29964.25 MB 2025-02-15 11:54:54,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30731.25 MB 2025-02-15 11:54:54,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:54:54,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36035.36 MB 2025-02-15 11:54:54,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36450.60 MB 2025-02-15 11:54:54,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 11:54:54,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31439.04 MB 2025-02-15 11:54:54,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:54:54,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:54:54,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:54:54,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31144.14 MB 2025-02-15 11:54:54,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31372.71 MB 2025-02-15 11:54:54,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-15 11:54:54,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36450.60 MB 2025-02-15 11:54:54,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36450.60 MB 2025-02-15 11:54:54,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:54:54,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31593.04 MB 2025-02-15 11:54:54,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:54:54,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:54:54,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.50 seconds 2025-02-15 11:54:54,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18323.74 MB 2025-02-15 11:54:54,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31573.19 MB 2025-02-15 11:54:54,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13249.46 MB 2025-02-15 11:54:54,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52808.38 MB 2025-02-15 11:54:54,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36450.60 MB 2025-02-15 11:54:54,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16357.79 MB 2025-02-15 11:54:54,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31593.04 MB 2025-02-15 11:54:54,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:54:54,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:54:54,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:54:54,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31573.19 MB 2025-02-15 11:54:54,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23319.35 MB 2025-02-15 11:54:54,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8253.84 MB 2025-02-15 11:54:54,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36450.60 MB 2025-02-15 11:54:54,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36450.60 MB 2025-02-15 11:54:54,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:54:54,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34077.49 MB 2025-02-15 11:54:54,948 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 11:54:54,949 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:54:54,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:54:54,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:54:54,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:54:54,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:54:54,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23319.35 MB 2025-02-15 11:54:54,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31733.33 MB 2025-02-15 11:54:54,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-15 11:54:54,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36450.60 MB 2025-02-15 11:54:54,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44816.14 MB 2025-02-15 11:54:54,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 11:54:54,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31733.33 MB 2025-02-15 11:54:55,117 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 11:54:55,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:55,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:54:55,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:55,119 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:54:55,124 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:54:55,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:54:55,125 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:54:55,125 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 11:55:51,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:51,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:55:51,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:55:51,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:51,123 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:55:51,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:51,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:55:54,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:55:54,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:55:54,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.03 seconds 2025-02-15 11:55:54,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:54,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-15 11:55:54,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.58 MB 2025-02-15 11:55:54,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 683.02 MB 2025-02-15 11:55:54,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57363.40 MB 2025-02-15 11:55:54,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17425.24 MB 2025-02-15 11:55:54,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39938.16 MB 2025-02-15 11:55:54,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24011.42 MB 2025-02-15 11:55:54,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:55:54,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:55:54,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:55:54,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:54,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.58 MB 2025-02-15 11:55:54,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15328.58 MB 2025-02-15 11:55:54,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 332.01 MB 2025-02-15 11:55:54,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17425.24 MB 2025-02-15 11:55:54,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19123.93 MB 2025-02-15 11:55:54,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1698.69 MB 2025-02-15 11:55:54,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17709.27 MB 2025-02-15 11:55:55,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:55:55,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:55:55,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 11:55:55,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15328.58 MB 2025-02-15 11:55:55,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15584.71 MB 2025-02-15 11:55:55,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 11:55:55,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19123.93 MB 2025-02-15 11:55:55,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18440.26 MB 2025-02-15 11:55:55,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -683.67 MB 2025-02-15 11:55:55,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19584.21 MB 2025-02-15 11:55:55,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:55:55,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:55:55,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:55:55,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15584.65 MB 2025-02-15 11:55:55,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16496.13 MB 2025-02-15 11:55:55,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 11:55:55,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18440.26 MB 2025-02-15 11:55:55,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18897.44 MB 2025-02-15 11:55:55,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-15 11:55:55,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17180.04 MB 2025-02-15 11:55:55,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:55:55,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:55:55,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 11:55:55,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16496.13 MB 2025-02-15 11:55:55,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.86 MB 2025-02-15 11:55:55,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 11:55:55,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18897.44 MB 2025-02-15 11:55:55,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21411.92 MB 2025-02-15 11:55:55,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2514.49 MB 2025-02-15 11:55:55,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20253.86 MB 2025-02-15 11:55:55,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:55:55,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:55:55,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 11:55:55,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15584.65 MB 2025-02-15 11:55:55,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.86 MB 2025-02-15 11:55:55,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 11:55:55,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18440.26 MB 2025-02-15 11:55:55,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21411.92 MB 2025-02-15 11:55:55,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2971.66 MB 2025-02-15 11:55:55,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20253.86 MB 2025-02-15 11:55:55,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:55:55,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:55:55,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:55:55,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18317.79 MB 2025-02-15 11:55:55,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18688.79 MB 2025-02-15 11:55:55,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.00 MB 2025-02-15 11:55:55,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21411.92 MB 2025-02-15 11:55:55,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21606.96 MB 2025-02-15 11:55:55,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-15 11:55:55,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19035.07 MB 2025-02-15 11:55:55,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:55:55,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:55:55,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:55:55,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18888.01 MB 2025-02-15 11:55:55,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19117.67 MB 2025-02-15 11:55:55,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.66 MB 2025-02-15 11:55:55,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21606.96 MB 2025-02-15 11:55:55,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21606.96 MB 2025-02-15 11:55:55,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:55:55,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19163.28 MB 2025-02-15 11:55:55,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:55:55,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:55:55,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.20 seconds 2025-02-15 11:55:55,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13641.13 MB 2025-02-15 11:55:55,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19318.74 MB 2025-02-15 11:55:55,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5677.61 MB 2025-02-15 11:55:55,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57363.40 MB 2025-02-15 11:55:55,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21606.96 MB 2025-02-15 11:55:55,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35756.44 MB 2025-02-15 11:55:55,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19318.74 MB 2025-02-15 11:55:55,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:55:55,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:55:55,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:55:55,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19318.74 MB 2025-02-15 11:55:55,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17669.28 MB 2025-02-15 11:55:55,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1649.46 MB 2025-02-15 11:55:55,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21606.96 MB 2025-02-15 11:55:55,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21606.96 MB 2025-02-15 11:55:55,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:55:55,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19318.74 MB 2025-02-15 11:55:55,610 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 11:55:55,610 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 11:55:55,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:55:55,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:55:55,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:55:55,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:55:55,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17669.28 MB 2025-02-15 11:55:55,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26108.31 MB 2025-02-15 11:55:55,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 11:55:55,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21606.96 MB 2025-02-15 11:55:55,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32096.91 MB 2025-02-15 11:55:55,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 11:55:55,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26108.31 MB 2025-02-15 11:55:55,777 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 11:55:55,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:55,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:55:55,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:55,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:55:55,784 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:55:55,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:55:55,785 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:55:55,785 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 11:57:28,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:28,825 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:57:28,830 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:57:28,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:28,834 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:57:28,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:28,835 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 11:57:48,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 11:57:48,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 11:57:48,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.41 seconds 2025-02-15 11:57:48,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:48,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-15 11:57:48,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-15 11:57:48,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-15 11:57:48,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44681.92 MB 2025-02-15 11:57:48,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37740.35 MB 2025-02-15 11:57:48,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6941.57 MB 2025-02-15 11:57:48,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-15 11:57:48,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 11:57:48,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 11:57:48,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 11:57:48,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:48,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-15 11:57:48,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-15 11:57:48,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-15 11:57:48,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37740.35 MB 2025-02-15 11:57:48,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46563.07 MB 2025-02-15 11:57:48,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8822.72 MB 2025-02-15 11:57:48,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39549.63 MB 2025-02-15 11:57:50,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 11:57:50,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 11:57:50,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 11:57:50,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-15 11:57:50,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-15 11:57:50,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 11:57:50,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46563.07 MB 2025-02-15 11:57:50,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 11:57:50,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17502.83 MB 2025-02-15 11:57:50,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26873.98 MB 2025-02-15 11:57:50,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 11:57:50,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 11:57:50,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 11:57:50,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 11:57:50,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-15 11:57:50,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 11:57:50,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:57:50,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29060.24 MB 2025-02-15 11:57:50,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:57:50,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-15 11:57:50,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 11:57:50,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 11:57:50,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 11:57:50,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-15 11:57:50,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 11:57:50,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 11:57:50,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:57:50,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:57:50,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 11:57:50,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 11:57:50,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 11:57:50,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 11:57:50,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 11:57:50,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 11:57:50,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 11:57:50,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 11:57:50,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29060.24 MB 2025-02-15 11:57:50,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 11:57:50,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 11:57:50,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 11:57:50,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 11:57:50,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 11:57:50,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 11:57:50,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-15 11:57:50,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-15 11:57:50,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 11:57:50,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 11:57:50,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 11:57:50,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 11:57:50,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-15 11:57:50,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 11:57:50,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 11:57:50,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:57:50,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-15 11:57:50,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29969.82 MB 2025-02-15 11:57:50,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.56 MB 2025-02-15 11:57:50,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 11:57:50,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 11:57:50,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:57:50,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30195.89 MB 2025-02-15 11:57:50,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 11:57:50,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 11:57:50,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.83 seconds 2025-02-15 11:57:50,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-15 11:57:50,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30170.37 MB 2025-02-15 11:57:50,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12787.34 MB 2025-02-15 11:57:50,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44681.92 MB 2025-02-15 11:57:50,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 11:57:50,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9539.94 MB 2025-02-15 11:57:50,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30195.89 MB 2025-02-15 11:57:50,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 11:57:50,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 11:57:50,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 11:57:50,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30170.37 MB 2025-02-15 11:57:50,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22379.72 MB 2025-02-15 11:57:50,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7790.65 MB 2025-02-15 11:57:50,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 11:57:50,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 11:57:50,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 11:57:50,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32675.59 MB 2025-02-15 11:57:50,957 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 11:57:50,958 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 11:57:50,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 11:57:50,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 11:57:50,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 11:57:50,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 11:57:50,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22379.72 MB 2025-02-15 11:57:50,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30796.84 MB 2025-02-15 11:57:50,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-15 11:57:50,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 11:57:50,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-15 11:57:50,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 11:57:50,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30796.84 MB 2025-02-15 11:57:51,125 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 11:57:51,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:51,126 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 11:57:51,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:51,127 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 11:57:51,132 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 11:57:51,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:57:51,133 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 11:57:51,133 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 11:59:37,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:59:37,517 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 11:59:37,522 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 11:59:37,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:59:37,526 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2100, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 11:59:37,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 11:59:37,527 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2100, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:00:09,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:00:09,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:00:09,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.31 seconds 2025-02-15 12:00:09,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:09,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27601.84 MB 2025-02-15 12:00:09,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35034.15 MB 2025-02-15 12:00:09,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7432.31 MB 2025-02-15 12:00:09,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47693.43 MB 2025-02-15 12:00:09,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40655.39 MB 2025-02-15 12:00:09,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7038.04 MB 2025-02-15 12:00:09,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43867.99 MB 2025-02-15 12:00:10,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:00:10,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:00:10,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:00:10,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:10,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35034.15 MB 2025-02-15 12:00:10,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26696.14 MB 2025-02-15 12:00:10,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8338.00 MB 2025-02-15 12:00:10,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40655.39 MB 2025-02-15 12:00:10,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56132.37 MB 2025-02-15 12:00:10,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15476.98 MB 2025-02-15 12:00:10,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55115.47 MB 2025-02-15 12:00:11,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:00:11,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:00:11,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:00:11,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:11,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26696.14 MB 2025-02-15 12:00:11,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27226.99 MB 2025-02-15 12:00:11,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:00:11,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56132.37 MB 2025-02-15 12:00:11,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31163.68 MB 2025-02-15 12:00:11,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24968.69 MB 2025-02-15 12:00:11,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31205.68 MB 2025-02-15 12:00:11,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:00:11,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:00:11,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:00:11,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:11,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27226.99 MB 2025-02-15 12:00:11,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.52 MB 2025-02-15 12:00:11,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:00:11,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31163.68 MB 2025-02-15 12:00:11,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33051.12 MB 2025-02-15 12:00:11,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:00:11,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30533.95 MB 2025-02-15 12:00:12,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:00:12,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:00:12,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:00:12,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29116.52 MB 2025-02-15 12:00:12,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31358.38 MB 2025-02-15 12:00:12,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:00:12,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33051.12 MB 2025-02-15 12:00:12,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38713.43 MB 2025-02-15 12:00:12,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:00:12,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36902.66 MB 2025-02-15 12:00:12,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:00:12,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:00:12,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:00:12,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27226.99 MB 2025-02-15 12:00:12,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31358.38 MB 2025-02-15 12:00:12,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:00:12,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31163.68 MB 2025-02-15 12:00:12,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38713.43 MB 2025-02-15 12:00:12,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:00:12,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36902.66 MB 2025-02-15 12:00:12,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:00:12,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:00:12,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:00:12,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32891.92 MB 2025-02-15 12:00:12,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33658.92 MB 2025-02-15 12:00:12,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:00:12,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38713.43 MB 2025-02-15 12:00:12,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39130.76 MB 2025-02-15 12:00:12,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:00:12,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34366.71 MB 2025-02-15 12:00:12,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:00:12,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:00:12,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:00:12,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34071.81 MB 2025-02-15 12:00:12,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34300.18 MB 2025-02-15 12:00:12,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-15 12:00:12,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39130.76 MB 2025-02-15 12:00:12,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39130.76 MB 2025-02-15 12:00:12,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:00:12,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34516.72 MB 2025-02-15 12:00:12,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:00:12,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:00:12,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.86 seconds 2025-02-15 12:00:12,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20285.27 MB 2025-02-15 12:00:12,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34501.03 MB 2025-02-15 12:00:12,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14215.76 MB 2025-02-15 12:00:12,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47693.43 MB 2025-02-15 12:00:12,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39130.76 MB 2025-02-15 12:00:12,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8562.67 MB 2025-02-15 12:00:12,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34516.72 MB 2025-02-15 12:00:12,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:00:12,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:00:12,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:00:12,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34501.03 MB 2025-02-15 12:00:12,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25278.04 MB 2025-02-15 12:00:12,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9222.99 MB 2025-02-15 12:00:12,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39130.76 MB 2025-02-15 12:00:12,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39130.76 MB 2025-02-15 12:00:12,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:00:12,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37002.87 MB 2025-02-15 12:00:12,677 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 12:00:12,677 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:00:12,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:00:12,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:00:12,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:00:12,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:00:12,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25278.04 MB 2025-02-15 12:00:12,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33683.70 MB 2025-02-15 12:00:12,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 12:00:12,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39130.76 MB 2025-02-15 12:00:12,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47490.01 MB 2025-02-15 12:00:12,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 12:00:12,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33683.70 MB 2025-02-15 12:00:12,845 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 12:00:12,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:12,847 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:00:12,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:12,848 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:00:12,852 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:00:12,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:12,853 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:00:12,853 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:00:25,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:25,732 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:00:25,736 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:00:25,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:25,740 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2489, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:00:25,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:00:25,741 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2489, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:01:04,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:01:04,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:01:04,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.88 seconds 2025-02-15 12:01:04,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:04,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.55 MB 2025-02-15 12:01:04,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39120.98 MB 2025-02-15 12:01:04,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8808.43 MB 2025-02-15 12:01:04,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73196.90 MB 2025-02-15 12:01:04,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42635.10 MB 2025-02-15 12:01:04,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30561.80 MB 2025-02-15 12:01:04,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47937.64 MB 2025-02-15 12:01:04,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:01:04,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:01:04,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 12:01:04,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:04,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39120.98 MB 2025-02-15 12:01:04,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28718.52 MB 2025-02-15 12:01:04,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10402.45 MB 2025-02-15 12:01:04,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42635.10 MB 2025-02-15 12:01:04,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61574.48 MB 2025-02-15 12:01:04,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18939.38 MB 2025-02-15 12:01:04,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64407.91 MB 2025-02-15 12:01:06,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:01:06,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:01:06,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 12:01:06,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:06,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28718.52 MB 2025-02-15 12:01:06,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29249.36 MB 2025-02-15 12:01:06,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:01:06,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61574.48 MB 2025-02-15 12:01:06,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31543.26 MB 2025-02-15 12:01:06,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30031.22 MB 2025-02-15 12:01:06,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33227.91 MB 2025-02-15 12:01:06,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:01:06,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:01:06,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:01:06,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:06,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29249.36 MB 2025-02-15 12:01:06,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31138.70 MB 2025-02-15 12:01:06,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.34 MB 2025-02-15 12:01:06,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31543.26 MB 2025-02-15 12:01:06,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34374.42 MB 2025-02-15 12:01:06,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:01:06,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32556.13 MB 2025-02-15 12:01:07,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:01:07,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:01:07,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:01:07,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31138.70 MB 2025-02-15 12:01:07,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33380.56 MB 2025-02-15 12:01:07,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:01:07,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34374.42 MB 2025-02-15 12:01:07,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40508.59 MB 2025-02-15 12:01:07,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:01:07,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38924.84 MB 2025-02-15 12:01:07,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:01:07,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:01:07,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:01:07,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29249.36 MB 2025-02-15 12:01:07,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33380.56 MB 2025-02-15 12:01:07,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.19 MB 2025-02-15 12:01:07,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31543.26 MB 2025-02-15 12:01:07,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40508.59 MB 2025-02-15 12:01:07,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 12:01:07,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38924.84 MB 2025-02-15 12:01:07,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:01:07,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:01:07,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:01:07,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34914.10 MB 2025-02-15 12:01:07,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35681.10 MB 2025-02-15 12:01:07,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:01:07,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40508.59 MB 2025-02-15 12:01:07,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 12:01:07,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:01:07,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36388.89 MB 2025-02-15 12:01:07,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:01:07,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:01:07,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:07,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36093.99 MB 2025-02-15 12:01:07,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36322.73 MB 2025-02-15 12:01:07,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-15 12:01:07,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-15 12:01:07,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 12:01:07,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:07,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36556.57 MB 2025-02-15 12:01:07,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:01:07,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:01:07,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.51 seconds 2025-02-15 12:01:07,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.63 MB 2025-02-15 12:01:07,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36523.39 MB 2025-02-15 12:01:07,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14882.76 MB 2025-02-15 12:01:07,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64523.08 MB 2025-02-15 12:01:07,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 12:01:07,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23597.15 MB 2025-02-15 12:01:07,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36556.57 MB 2025-02-15 12:01:07,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:01:07,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:01:07,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:01:07,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36523.39 MB 2025-02-15 12:01:07,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26638.74 MB 2025-02-15 12:01:07,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9884.65 MB 2025-02-15 12:01:07,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-15 12:01:07,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-15 12:01:07,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:07,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39029.83 MB 2025-02-15 12:01:07,548 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 12:01:07,548 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:01:07,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:01:07,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:01:07,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:07,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:07,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26638.74 MB 2025-02-15 12:01:07,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35060.03 MB 2025-02-15 12:01:07,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.30 MB 2025-02-15 12:01:07,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-15 12:01:07,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45111.84 MB 2025-02-15 12:01:07,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 12:01:07,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35060.03 MB 2025-02-15 12:01:07,717 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 12:01:07,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:07,718 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:01:07,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:07,719 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:01:07,724 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:01:07,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:07,725 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:01:07,725 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:01:17,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:17,860 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:01:17,865 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:01:17,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:17,869 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:01:17,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:17,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:01:21,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:01:21,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:01:21,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.77 seconds 2025-02-15 12:01:21,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:21,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-15 12:01:21,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15427.37 MB 2025-02-15 12:01:21,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-15 12:01:21,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53483.67 MB 2025-02-15 12:01:21,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-15 12:01:21,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33244.05 MB 2025-02-15 12:01:21,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.12 MB 2025-02-15 12:01:21,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:01:21,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:01:21,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:21,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:21,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15427.37 MB 2025-02-15 12:01:21,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15765.31 MB 2025-02-15 12:01:21,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.95 MB 2025-02-15 12:01:21,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-15 12:01:21,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-15 12:01:21,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:21,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18587.73 MB 2025-02-15 12:01:22,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:01:22,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:01:22,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.13 seconds 2025-02-15 12:01:22,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:22,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15765.31 MB 2025-02-15 12:01:22,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16063.91 MB 2025-02-15 12:01:22,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.60 MB 2025-02-15 12:01:22,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-15 12:01:22,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18601.74 MB 2025-02-15 12:01:22,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 12:01:22,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20020.94 MB 2025-02-15 12:01:22,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:01:22,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:01:22,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:01:22,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:22,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16063.91 MB 2025-02-15 12:01:22,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17126.52 MB 2025-02-15 12:01:22,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1062.60 MB 2025-02-15 12:01:22,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18601.74 MB 2025-02-15 12:01:22,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-15 12:01:22,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1065.35 MB 2025-02-15 12:01:22,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17923.83 MB 2025-02-15 12:01:22,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:01:22,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:01:22,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:01:22,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:22,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17126.52 MB 2025-02-15 12:01:22,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18387.59 MB 2025-02-15 12:01:22,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.07 MB 2025-02-15 12:01:22,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-15 12:01:22,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22863.15 MB 2025-02-15 12:01:22,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3196.06 MB 2025-02-15 12:01:22,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21508.06 MB 2025-02-15 12:01:22,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:01:22,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:01:22,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:01:22,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:22,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16063.91 MB 2025-02-15 12:01:22,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18387.59 MB 2025-02-15 12:01:22,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2323.68 MB 2025-02-15 12:01:22,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18601.74 MB 2025-02-15 12:01:22,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22863.15 MB 2025-02-15 12:01:22,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4261.41 MB 2025-02-15 12:01:22,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21508.06 MB 2025-02-15 12:01:23,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:01:23,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:01:23,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:01:23,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:23,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19250.21 MB 2025-02-15 12:01:23,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19683.48 MB 2025-02-15 12:01:23,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 433.27 MB 2025-02-15 12:01:23,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22863.15 MB 2025-02-15 12:01:23,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23098.03 MB 2025-02-15 12:01:23,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 234.88 MB 2025-02-15 12:01:23,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20082.10 MB 2025-02-15 12:01:23,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:01:23,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:01:23,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:23,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:23,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19915.74 MB 2025-02-15 12:01:23,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20140.67 MB 2025-02-15 12:01:23,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.94 MB 2025-02-15 12:01:23,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23098.03 MB 2025-02-15 12:01:23,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23098.03 MB 2025-02-15 12:01:23,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:23,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20198.30 MB 2025-02-15 12:01:23,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:01:23,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:01:23,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.29 seconds 2025-02-15 12:01:23,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:23,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13783.98 MB 2025-02-15 12:01:23,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20341.75 MB 2025-02-15 12:01:23,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6557.77 MB 2025-02-15 12:01:23,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53483.67 MB 2025-02-15 12:01:23,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23098.03 MB 2025-02-15 12:01:23,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30385.64 MB 2025-02-15 12:01:23,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20341.75 MB 2025-02-15 12:01:23,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:01:23,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:01:23,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:01:23,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:23,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14949.84 MB 2025-02-15 12:01:23,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17963.88 MB 2025-02-15 12:01:23,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 12:01:23,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23098.03 MB 2025-02-15 12:01:23,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23098.03 MB 2025-02-15 12:01:23,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:23,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18265.25 MB 2025-02-15 12:01:23,477 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:01:23,477 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:01:23,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:01:23,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:01:23,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:23,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:23,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17963.88 MB 2025-02-15 12:01:23,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26402.90 MB 2025-02-15 12:01:23,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:01:23,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23098.03 MB 2025-02-15 12:01:23,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33587.99 MB 2025-02-15 12:01:23,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:01:23,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26402.90 MB 2025-02-15 12:01:23,737 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:01:23,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:23,740 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:01:23,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:23,742 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:01:23,749 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:01:23,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:23,751 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:01:23,752 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:01:37,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:37,317 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:01:37,322 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:01:37,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:37,326 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:01:37,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:37,327 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:01:39,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:01:39,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:01:39,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.43 seconds 2025-02-15 12:01:39,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:39,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-15 12:01:39,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-15 12:01:39,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 12:01:39,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46173.00 MB 2025-02-15 12:01:39,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17716.74 MB 2025-02-15 12:01:39,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28456.26 MB 2025-02-15 12:01:39,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-15 12:01:39,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:01:39,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:01:39,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:01:39,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:39,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-15 12:01:39,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14804.04 MB 2025-02-15 12:01:39,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.74 MB 2025-02-15 12:01:39,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17716.74 MB 2025-02-15 12:01:39,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18241.03 MB 2025-02-15 12:01:39,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 524.29 MB 2025-02-15 12:01:39,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16655.70 MB 2025-02-15 12:01:40,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:01:40,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:01:40,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 12:01:40,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14804.04 MB 2025-02-15 12:01:40,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15000.45 MB 2025-02-15 12:01:40,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-15 12:01:40,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18241.03 MB 2025-02-15 12:01:40,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18241.03 MB 2025-02-15 12:01:40,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:40,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18975.76 MB 2025-02-15 12:01:40,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:01:40,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:01:40,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:01:40,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15000.38 MB 2025-02-15 12:01:40,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15699.34 MB 2025-02-15 12:01:40,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-15 12:01:40,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18241.03 MB 2025-02-15 12:01:40,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18241.03 MB 2025-02-15 12:01:40,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:40,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16223.80 MB 2025-02-15 12:01:40,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:01:40,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:01:40,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:01:40,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.34 MB 2025-02-15 12:01:40,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.87 MB 2025-02-15 12:01:40,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-15 12:01:40,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18241.03 MB 2025-02-15 12:01:40,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19467.86 MB 2025-02-15 12:01:40,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1226.83 MB 2025-02-15 12:01:40,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18583.36 MB 2025-02-15 12:01:40,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:01:40,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:01:40,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:01:40,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15000.38 MB 2025-02-15 12:01:40,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.87 MB 2025-02-15 12:01:40,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-15 12:01:40,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18241.03 MB 2025-02-15 12:01:40,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19467.86 MB 2025-02-15 12:01:40,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1226.83 MB 2025-02-15 12:01:40,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18583.36 MB 2025-02-15 12:01:40,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:01:40,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:01:40,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:01:40,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17096.28 MB 2025-02-15 12:01:40,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17380.07 MB 2025-02-15 12:01:40,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-15 12:01:40,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19467.86 MB 2025-02-15 12:01:40,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19623.05 MB 2025-02-15 12:01:40,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 155.19 MB 2025-02-15 12:01:40,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17651.44 MB 2025-02-15 12:01:40,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:01:40,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:01:40,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:01:40,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17532.85 MB 2025-02-15 12:01:40,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17746.50 MB 2025-02-15 12:01:40,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.65 MB 2025-02-15 12:01:40,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19623.05 MB 2025-02-15 12:01:40,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19623.05 MB 2025-02-15 12:01:40,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:01:40,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17749.25 MB 2025-02-15 12:01:40,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:01:40,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:01:40,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.34 seconds 2025-02-15 12:01:40,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-15 12:01:40,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17947.57 MB 2025-02-15 12:01:40,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4445.80 MB 2025-02-15 12:01:40,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46173.00 MB 2025-02-15 12:01:40,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19623.05 MB 2025-02-15 12:01:40,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26549.94 MB 2025-02-15 12:01:40,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17947.57 MB 2025-02-15 12:01:40,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:01:40,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:01:40,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:01:40,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17947.57 MB 2025-02-15 12:01:40,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17316.37 MB 2025-02-15 12:01:40,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -631.19 MB 2025-02-15 12:01:40,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19623.05 MB 2025-02-15 12:01:40,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19757.27 MB 2025-02-15 12:01:40,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-15 12:01:40,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19052.96 MB 2025-02-15 12:01:40,953 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:01:40,953 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:01:40,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:01:40,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:01:40,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:01:40,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:01:40,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17316.37 MB 2025-02-15 12:01:40,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25755.40 MB 2025-02-15 12:01:40,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:01:40,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19757.27 MB 2025-02-15 12:01:40,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30247.22 MB 2025-02-15 12:01:40,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:01:40,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25755.40 MB 2025-02-15 12:01:41,117 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:01:41,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:41,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:01:41,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:41,119 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:01:41,124 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:01:41,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:01:41,125 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:01:41,125 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:02:27,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:27,282 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:02:27,287 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:02:27,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:27,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 233, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:02:27,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:27,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 233, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:02:30,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:02:30,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:02:30,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.60 seconds 2025-02-15 12:02:30,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:30,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14592.29 MB 2025-02-15 12:02:30,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15416.86 MB 2025-02-15 12:02:30,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 824.57 MB 2025-02-15 12:02:30,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42832.23 MB 2025-02-15 12:02:30,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21447.57 MB 2025-02-15 12:02:30,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21384.66 MB 2025-02-15 12:02:30,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24290.15 MB 2025-02-15 12:02:30,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:02:30,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:02:30,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:02:30,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:30,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15416.86 MB 2025-02-15 12:02:30,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.05 MB 2025-02-15 12:02:30,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.19 MB 2025-02-15 12:02:30,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21447.57 MB 2025-02-15 12:02:30,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21447.57 MB 2025-02-15 12:02:30,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:02:30,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18535.08 MB 2025-02-15 12:02:31,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:02:31,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:02:31,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.06 seconds 2025-02-15 12:02:31,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:31,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15739.05 MB 2025-02-15 12:02:31,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16033.66 MB 2025-02-15 12:02:31,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-15 12:02:31,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21447.57 MB 2025-02-15 12:02:31,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19809.70 MB 2025-02-15 12:02:31,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-15 12:02:31,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19994.67 MB 2025-02-15 12:02:31,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:02:31,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:02:31,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:02:31,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:31,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16033.66 MB 2025-02-15 12:02:31,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17082.10 MB 2025-02-15 12:02:31,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-15 12:02:31,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19809.70 MB 2025-02-15 12:02:31,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19809.70 MB 2025-02-15 12:02:31,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:02:31,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17868.78 MB 2025-02-15 12:02:32,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:02:32,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:02:32,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:02:32,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17082.10 MB 2025-02-15 12:02:32,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18326.36 MB 2025-02-15 12:02:32,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.26 MB 2025-02-15 12:02:32,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19809.70 MB 2025-02-15 12:02:32,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22955.43 MB 2025-02-15 12:02:32,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3145.73 MB 2025-02-15 12:02:32,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21406.03 MB 2025-02-15 12:02:32,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:02:32,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:02:32,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:02:32,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16033.66 MB 2025-02-15 12:02:32,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18326.36 MB 2025-02-15 12:02:32,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2292.70 MB 2025-02-15 12:02:32,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19809.70 MB 2025-02-15 12:02:32,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22955.43 MB 2025-02-15 12:02:32,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3145.73 MB 2025-02-15 12:02:32,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21406.03 MB 2025-02-15 12:02:32,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:02:32,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:02:32,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:02:32,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19177.48 MB 2025-02-15 12:02:32,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19603.69 MB 2025-02-15 12:02:32,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 426.21 MB 2025-02-15 12:02:32,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22955.43 MB 2025-02-15 12:02:32,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23186.11 MB 2025-02-15 12:02:32,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 230.69 MB 2025-02-15 12:02:32,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19996.51 MB 2025-02-15 12:02:32,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:02:32,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:02:32,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:02:32,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19832.84 MB 2025-02-15 12:02:32,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20051.77 MB 2025-02-15 12:02:32,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.93 MB 2025-02-15 12:02:32,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23186.11 MB 2025-02-15 12:02:32,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23186.11 MB 2025-02-15 12:02:32,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:02:32,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20127.22 MB 2025-02-15 12:02:32,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:02:32,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:02:32,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.92 seconds 2025-02-15 12:02:32,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13780.50 MB 2025-02-15 12:02:32,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20252.68 MB 2025-02-15 12:02:32,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6472.18 MB 2025-02-15 12:02:32,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42832.23 MB 2025-02-15 12:02:32,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23186.11 MB 2025-02-15 12:02:32,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19646.12 MB 2025-02-15 12:02:32,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20252.68 MB 2025-02-15 12:02:32,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:02:32,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:02:32,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:02:32,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14931.26 MB 2025-02-15 12:02:32,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17942.71 MB 2025-02-15 12:02:32,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.45 MB 2025-02-15 12:02:32,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23186.11 MB 2025-02-15 12:02:32,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23186.11 MB 2025-02-15 12:02:32,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:02:32,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18243.82 MB 2025-02-15 12:02:32,507 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 12:02:32,507 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:02:32,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:02:32,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:02:32,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:02:32,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:02:32,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17942.71 MB 2025-02-15 12:02:32,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26374.17 MB 2025-02-15 12:02:32,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 12:02:32,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23186.11 MB 2025-02-15 12:02:32,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31570.53 MB 2025-02-15 12:02:32,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 12:02:32,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26374.17 MB 2025-02-15 12:02:32,671 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 12:02:32,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:32,672 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:02:32,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:32,673 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:02:32,678 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:02:32,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:02:32,679 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:02:32,679 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:03:50,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:03:50,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:03:50,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:03:50,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:03:50,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1069, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:03:50,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:03:50,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1069, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:04:07,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:04:07,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:04:07,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.40 seconds 2025-02-15 12:04:07,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:07,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20417.67 MB 2025-02-15 12:04:07,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24200.93 MB 2025-02-15 12:04:07,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3783.26 MB 2025-02-15 12:04:07,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39954.94 MB 2025-02-15 12:04:07,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26549.94 MB 2025-02-15 12:04:07,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13405.00 MB 2025-02-15 12:04:07,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33060.74 MB 2025-02-15 12:04:07,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:04:07,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:04:07,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:04:07,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:07,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24200.93 MB 2025-02-15 12:04:07,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21336.30 MB 2025-02-15 12:04:07,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2864.63 MB 2025-02-15 12:04:07,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26549.94 MB 2025-02-15 12:04:07,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34682.70 MB 2025-02-15 12:04:07,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8132.76 MB 2025-02-15 12:04:07,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34431.84 MB 2025-02-15 12:04:09,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:04:09,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:04:09,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:04:09,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21336.30 MB 2025-02-15 12:04:09,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21867.14 MB 2025-02-15 12:04:09,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:04:09,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34682.70 MB 2025-02-15 12:04:09,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24891.10 MB 2025-02-15 12:04:09,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9791.60 MB 2025-02-15 12:04:09,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25846.72 MB 2025-02-15 12:04:09,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:04:09,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:04:09,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:04:09,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21867.14 MB 2025-02-15 12:04:09,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23756.67 MB 2025-02-15 12:04:09,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:04:09,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24891.10 MB 2025-02-15 12:04:09,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27722.25 MB 2025-02-15 12:04:09,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:04:09,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25174.10 MB 2025-02-15 12:04:09,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:04:09,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:04:09,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:04:09,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23756.67 MB 2025-02-15 12:04:09,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25998.53 MB 2025-02-15 12:04:09,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:04:09,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27722.25 MB 2025-02-15 12:04:09,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33858.52 MB 2025-02-15 12:04:09,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 12:04:09,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31542.81 MB 2025-02-15 12:04:09,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:04:09,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:04:09,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:04:09,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21867.14 MB 2025-02-15 12:04:09,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25998.53 MB 2025-02-15 12:04:09,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:04:09,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24891.10 MB 2025-02-15 12:04:09,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33858.52 MB 2025-02-15 12:04:09,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8967.42 MB 2025-02-15 12:04:09,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31542.81 MB 2025-02-15 12:04:09,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:04:09,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:04:09,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:04:09,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27532.07 MB 2025-02-15 12:04:09,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28299.07 MB 2025-02-15 12:04:09,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:04:09,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33858.52 MB 2025-02-15 12:04:09,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34275.85 MB 2025-02-15 12:04:09,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:04:09,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29006.86 MB 2025-02-15 12:04:09,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:04:09,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:04:09,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:04:09,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28711.96 MB 2025-02-15 12:04:09,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28941.68 MB 2025-02-15 12:04:09,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.72 MB 2025-02-15 12:04:09,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34275.85 MB 2025-02-15 12:04:09,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34275.85 MB 2025-02-15 12:04:09,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:04:09,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29156.92 MB 2025-02-15 12:04:09,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:04:09,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:04:09,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.83 seconds 2025-02-15 12:04:09,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:09,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16693.19 MB 2025-02-15 12:04:09,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29142.76 MB 2025-02-15 12:04:09,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12449.57 MB 2025-02-15 12:04:09,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39954.94 MB 2025-02-15 12:04:09,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34275.85 MB 2025-02-15 12:04:09,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5679.09 MB 2025-02-15 12:04:09,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29156.92 MB 2025-02-15 12:04:10,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:04:10,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:04:10,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:04:10,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:10,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29142.76 MB 2025-02-15 12:04:10,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21697.58 MB 2025-02-15 12:04:10,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7445.18 MB 2025-02-15 12:04:10,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34275.85 MB 2025-02-15 12:04:10,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34275.85 MB 2025-02-15 12:04:10,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:04:10,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31654.42 MB 2025-02-15 12:04:10,027 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:04:10,027 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:04:10,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:04:10,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:04:10,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:04:10,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:04:10,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21697.58 MB 2025-02-15 12:04:10,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30136.60 MB 2025-02-15 12:04:10,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:04:10,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34275.85 MB 2025-02-15 12:04:10,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44765.81 MB 2025-02-15 12:04:10,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:04:10,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30136.60 MB 2025-02-15 12:04:10,192 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:04:10,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:04:10,193 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:04:10,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:04:10,194 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:04:10,199 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:04:10,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:04:10,200 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:04:10,200 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:05:02,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:02,522 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:05:02,527 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:05:02,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:02,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:05:02,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:02,532 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:05:29,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:05:29,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:05:29,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.01 seconds 2025-02-15 12:05:29,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:29,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.86 MB 2025-02-15 12:05:29,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31398.43 MB 2025-02-15 12:05:29,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-15 12:05:29,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57350.82 MB 2025-02-15 12:05:29,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39462.11 MB 2025-02-15 12:05:29,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17888.71 MB 2025-02-15 12:05:29,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40324.54 MB 2025-02-15 12:05:29,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:05:29,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:05:29,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:05:29,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:29,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31398.43 MB 2025-02-15 12:05:29,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24896.35 MB 2025-02-15 12:05:29,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.08 MB 2025-02-15 12:05:29,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39462.11 MB 2025-02-15 12:05:29,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52722.40 MB 2025-02-15 12:05:29,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13260.29 MB 2025-02-15 12:05:29,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49207.40 MB 2025-02-15 12:05:31,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:05:31,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:05:31,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:05:31,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:31,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24896.35 MB 2025-02-15 12:05:31,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25427.19 MB 2025-02-15 12:05:31,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:05:31,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52722.40 MB 2025-02-15 12:05:31,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-15 12:05:31,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18052.28 MB 2025-02-15 12:05:31,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.74 MB 2025-02-15 12:05:31,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:05:31,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:05:31,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:05:31,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:31,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-15 12:05:31,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27316.72 MB 2025-02-15 12:05:31,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:05:31,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 12:05:31,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-15 12:05:31,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:05:31,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28734.15 MB 2025-02-15 12:05:31,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:05:31,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:05:31,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:05:31,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:31,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27316.72 MB 2025-02-15 12:05:31,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-15 12:05:31,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:05:31,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 12:05:31,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37501.27 MB 2025-02-15 12:05:31,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:05:31,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-15 12:05:31,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:05:31,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:05:31,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:05:31,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:31,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-15 12:05:31,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-15 12:05:31,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:05:31,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-15 12:05:31,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37501.27 MB 2025-02-15 12:05:31,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:05:31,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-15 12:05:32,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:05:32,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:05:32,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:05:32,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:32,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31092.12 MB 2025-02-15 12:05:32,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31859.12 MB 2025-02-15 12:05:32,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:05:32,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37501.27 MB 2025-02-15 12:05:32,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-15 12:05:32,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:05:32,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32566.91 MB 2025-02-15 12:05:32,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:05:32,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:05:32,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:05:32,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:32,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32272.01 MB 2025-02-15 12:05:32,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32500.53 MB 2025-02-15 12:05:32,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-15 12:05:32,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-15 12:05:32,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-15 12:05:32,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:05:32,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32708.19 MB 2025-02-15 12:05:32,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:05:32,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:05:32,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.56 seconds 2025-02-15 12:05:32,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:32,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19079.78 MB 2025-02-15 12:05:32,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32700.97 MB 2025-02-15 12:05:32,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13621.18 MB 2025-02-15 12:05:32,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57350.82 MB 2025-02-15 12:05:32,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-15 12:05:32,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19434.31 MB 2025-02-15 12:05:32,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32708.19 MB 2025-02-15 12:05:32,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:05:32,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:05:32,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:05:32,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:32,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32700.97 MB 2025-02-15 12:05:32,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24074.69 MB 2025-02-15 12:05:32,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8626.28 MB 2025-02-15 12:05:32,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-15 12:05:32,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-15 12:05:32,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:05:32,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35205.06 MB 2025-02-15 12:05:32,401 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 12:05:32,402 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:05:32,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:05:32,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:05:32,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:05:32,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:05:32,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24074.69 MB 2025-02-15 12:05:32,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32486.54 MB 2025-02-15 12:05:32,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8411.85 MB 2025-02-15 12:05:32,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-15 12:05:32,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42098.23 MB 2025-02-15 12:05:32,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-15 12:05:32,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32486.54 MB 2025-02-15 12:05:32,659 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 12:05:32,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:32,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:05:32,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:32,664 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:05:32,671 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:05:32,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:05:32,673 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:05:32,673 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:06:23,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:23,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:06:23,241 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:06:23,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:23,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:06:23,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:23,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:06:42,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:06:42,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:06:42,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.43 seconds 2025-02-15 12:06:42,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:42,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-15 12:06:42,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-15 12:06:42,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-15 12:06:42,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50461.67 MB 2025-02-15 12:06:42,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37669.04 MB 2025-02-15 12:06:42,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12792.63 MB 2025-02-15 12:06:42,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-15 12:06:42,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:06:42,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:06:42,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:06:42,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:42,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-15 12:06:42,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-15 12:06:42,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-15 12:06:42,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37669.04 MB 2025-02-15 12:06:42,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46409.97 MB 2025-02-15 12:06:42,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8740.93 MB 2025-02-15 12:06:42,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39316.05 MB 2025-02-15 12:06:44,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:06:44,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:06:44,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:06:44,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:44,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-15 12:06:44,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-15 12:06:44,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:06:44,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46409.97 MB 2025-02-15 12:06:44,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29037.17 MB 2025-02-15 12:06:44,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17372.81 MB 2025-02-15 12:06:44,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.99 MB 2025-02-15 12:06:44,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:06:44,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:06:44,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:06:44,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:44,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-15 12:06:44,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-15 12:06:44,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:06:44,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 12:06:44,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29037.17 MB 2025-02-15 12:06:44,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:06:44,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-15 12:06:44,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:06:44,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:06:44,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:06:44,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:44,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-15 12:06:44,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-15 12:06:44,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:06:44,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 12:06:44,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34699.48 MB 2025-02-15 12:06:44,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:06:44,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-15 12:06:44,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:06:44,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:06:44,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:06:44,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:44,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-15 12:06:44,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-15 12:06:44,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:06:44,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 12:06:44,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34699.48 MB 2025-02-15 12:06:44,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:06:44,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-15 12:06:45,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:06:45,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:06:45,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:06:45,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:45,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-15 12:06:45,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-15 12:06:45,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:06:45,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34699.48 MB 2025-02-15 12:06:45,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35112.62 MB 2025-02-15 12:06:45,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 12:06:45,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-15 12:06:45,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:06:45,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:06:45,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:06:45,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:45,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-15 12:06:45,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29917.01 MB 2025-02-15 12:06:45,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-15 12:06:45,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35112.62 MB 2025-02-15 12:06:45,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35112.62 MB 2025-02-15 12:06:45,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:06:45,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30152.56 MB 2025-02-15 12:06:45,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:06:45,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:06:45,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.86 seconds 2025-02-15 12:06:45,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:45,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-15 12:06:45,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30117.66 MB 2025-02-15 12:06:45,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12769.47 MB 2025-02-15 12:06:45,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50461.67 MB 2025-02-15 12:06:45,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35112.62 MB 2025-02-15 12:06:45,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15349.06 MB 2025-02-15 12:06:45,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30152.56 MB 2025-02-15 12:06:45,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:06:45,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:06:45,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:06:45,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:45,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30117.66 MB 2025-02-15 12:06:45,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22346.31 MB 2025-02-15 12:06:45,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7771.36 MB 2025-02-15 12:06:45,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35112.62 MB 2025-02-15 12:06:45,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35112.62 MB 2025-02-15 12:06:45,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:06:45,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32624.11 MB 2025-02-15 12:06:45,393 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 12:06:45,393 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:06:45,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:06:45,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:06:45,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:06:45,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:06:45,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22346.31 MB 2025-02-15 12:06:45,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30767.60 MB 2025-02-15 12:06:45,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.30 MB 2025-02-15 12:06:45,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35112.62 MB 2025-02-15 12:06:45,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39298.53 MB 2025-02-15 12:06:45,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 12:06:45,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30767.60 MB 2025-02-15 12:06:45,556 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 12:06:45,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:45,558 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:06:45,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:45,559 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:06:45,563 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:06:45,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:06:45,564 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:06:45,565 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:07:21,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:21,322 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:07:21,327 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:07:21,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:21,330 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1277, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:07:21,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:21,331 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1277, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:07:41,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:07:41,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:07:41,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.75 seconds 2025-02-15 12:07:41,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:41,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21867.05 MB 2025-02-15 12:07:41,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26386.41 MB 2025-02-15 12:07:41,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4519.36 MB 2025-02-15 12:07:41,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47670.36 MB 2025-02-15 12:07:41,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37731.96 MB 2025-02-15 12:07:41,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9938.40 MB 2025-02-15 12:07:41,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35188.79 MB 2025-02-15 12:07:41,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:07:41,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:07:41,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:07:41,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:41,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26386.41 MB 2025-02-15 12:07:41,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22416.58 MB 2025-02-15 12:07:41,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3969.83 MB 2025-02-15 12:07:41,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37731.96 MB 2025-02-15 12:07:41,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-15 12:07:41,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8803.84 MB 2025-02-15 12:07:41,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39620.60 MB 2025-02-15 12:07:43,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:07:43,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:07:43,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 12:07:43,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22416.58 MB 2025-02-15 12:07:43,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22947.42 MB 2025-02-15 12:07:43,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:07:43,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46535.80 MB 2025-02-15 12:07:43,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33212.60 MB 2025-02-15 12:07:43,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13323.21 MB 2025-02-15 12:07:43,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26925.96 MB 2025-02-15 12:07:43,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:07:43,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:07:43,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:07:43,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22947.42 MB 2025-02-15 12:07:43,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24836.95 MB 2025-02-15 12:07:43,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:07:43,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33212.60 MB 2025-02-15 12:07:43,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33212.60 MB 2025-02-15 12:07:43,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:07:43,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26254.38 MB 2025-02-15 12:07:43,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:07:43,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:07:43,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:07:43,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24836.95 MB 2025-02-15 12:07:43,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.81 MB 2025-02-15 12:07:43,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:07:43,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33212.60 MB 2025-02-15 12:07:43,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34156.31 MB 2025-02-15 12:07:43,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:07:43,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32623.09 MB 2025-02-15 12:07:43,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:07:43,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:07:43,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:07:43,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22947.42 MB 2025-02-15 12:07:43,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.81 MB 2025-02-15 12:07:43,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:07:43,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33212.60 MB 2025-02-15 12:07:43,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34156.31 MB 2025-02-15 12:07:43,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:07:43,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32623.09 MB 2025-02-15 12:07:43,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:07:43,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:07:43,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 12:07:43,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28612.35 MB 2025-02-15 12:07:43,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29379.35 MB 2025-02-15 12:07:43,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:07:43,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34156.31 MB 2025-02-15 12:07:43,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 12:07:43,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:07:43,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30087.14 MB 2025-02-15 12:07:43,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:07:43,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:07:43,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:07:43,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29792.24 MB 2025-02-15 12:07:43,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30019.98 MB 2025-02-15 12:07:43,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.74 MB 2025-02-15 12:07:43,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 12:07:43,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 12:07:43,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:07:43,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30247.43 MB 2025-02-15 12:07:43,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:07:43,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:07:43,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.23 seconds 2025-02-15 12:07:43,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17417.88 MB 2025-02-15 12:07:43,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30220.84 MB 2025-02-15 12:07:43,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12802.96 MB 2025-02-15 12:07:43,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47670.36 MB 2025-02-15 12:07:43,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 12:07:43,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13096.71 MB 2025-02-15 12:07:43,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30247.43 MB 2025-02-15 12:07:43,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:07:43,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:07:43,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:07:43,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30220.84 MB 2025-02-15 12:07:43,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22409.57 MB 2025-02-15 12:07:43,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7811.26 MB 2025-02-15 12:07:43,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 12:07:43,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34573.65 MB 2025-02-15 12:07:43,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:07:43,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32721.75 MB 2025-02-15 12:07:43,851 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-15 12:07:43,852 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:07:43,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:07:43,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:07:43,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:07:43,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:07:43,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22409.57 MB 2025-02-15 12:07:43,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30812.14 MB 2025-02-15 12:07:43,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8402.56 MB 2025-02-15 12:07:43,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34573.65 MB 2025-02-15 12:07:43,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38751.17 MB 2025-02-15 12:07:43,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 12:07:43,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30812.14 MB 2025-02-15 12:07:44,020 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-15 12:07:44,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:44,021 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:07:44,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:44,022 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:07:44,027 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:07:44,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:07:44,028 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:07:44,028 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:09:19,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:19,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:09:19,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:09:19,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:19,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 972, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:09:19,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:19,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 972, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:09:34,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:09:34,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:09:34,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.93 seconds 2025-02-15 12:09:34,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:34,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19741.76 MB 2025-02-15 12:09:34,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23181.61 MB 2025-02-15 12:09:34,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3439.85 MB 2025-02-15 12:09:34,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-15 12:09:34,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28292.68 MB 2025-02-15 12:09:34,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18813.55 MB 2025-02-15 12:09:34,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32157.53 MB 2025-02-15 12:09:34,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:09:34,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:09:34,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:09:34,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:34,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.61 MB 2025-02-15 12:09:34,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20832.02 MB 2025-02-15 12:09:34,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2349.59 MB 2025-02-15 12:09:34,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28292.68 MB 2025-02-15 12:09:34,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35104.23 MB 2025-02-15 12:09:34,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6811.55 MB 2025-02-15 12:09:34,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32093.18 MB 2025-02-15 12:09:36,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:09:36,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:09:36,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:09:36,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20832.02 MB 2025-02-15 12:09:36,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21362.87 MB 2025-02-15 12:09:36,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:09:36,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35104.23 MB 2025-02-15 12:09:36,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26975.67 MB 2025-02-15 12:09:36,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8128.56 MB 2025-02-15 12:09:36,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25341.41 MB 2025-02-15 12:09:36,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:09:36,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:09:36,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:09:36,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.87 MB 2025-02-15 12:09:36,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23252.40 MB 2025-02-15 12:09:36,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:09:36,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26975.67 MB 2025-02-15 12:09:36,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26975.67 MB 2025-02-15 12:09:36,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:09:36,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24669.83 MB 2025-02-15 12:09:36,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:09:36,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:09:36,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:09:36,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23252.40 MB 2025-02-15 12:09:36,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25494.26 MB 2025-02-15 12:09:36,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:09:36,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26975.67 MB 2025-02-15 12:09:36,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33109.84 MB 2025-02-15 12:09:36,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:09:36,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31038.54 MB 2025-02-15 12:09:36,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:09:36,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:09:36,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:09:36,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.87 MB 2025-02-15 12:09:36,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25494.26 MB 2025-02-15 12:09:36,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:09:36,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26975.67 MB 2025-02-15 12:09:36,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33109.84 MB 2025-02-15 12:09:36,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:09:36,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31038.54 MB 2025-02-15 12:09:36,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:09:36,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:09:36,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:09:36,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27027.80 MB 2025-02-15 12:09:36,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27794.80 MB 2025-02-15 12:09:36,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:09:36,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33109.84 MB 2025-02-15 12:09:36,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33525.07 MB 2025-02-15 12:09:36,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:09:36,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28502.59 MB 2025-02-15 12:09:36,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:09:36,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:09:36,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:09:36,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28207.69 MB 2025-02-15 12:09:36,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28438.70 MB 2025-02-15 12:09:36,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.01 MB 2025-02-15 12:09:36,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33525.07 MB 2025-02-15 12:09:36,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33525.07 MB 2025-02-15 12:09:36,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:09:36,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.75 MB 2025-02-15 12:09:36,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:09:36,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:09:36,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.32 seconds 2025-02-15 12:09:36,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16355.23 MB 2025-02-15 12:09:36,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28639.78 MB 2025-02-15 12:09:36,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12284.54 MB 2025-02-15 12:09:36,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-15 12:09:36,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33525.07 MB 2025-02-15 12:09:36,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13581.16 MB 2025-02-15 12:09:36,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.75 MB 2025-02-15 12:09:36,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:09:36,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:09:36,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:09:36,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28639.78 MB 2025-02-15 12:09:36,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21359.62 MB 2025-02-15 12:09:36,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7280.15 MB 2025-02-15 12:09:36,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33525.07 MB 2025-02-15 12:09:36,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33525.07 MB 2025-02-15 12:09:36,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:09:36,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31151.44 MB 2025-02-15 12:09:36,733 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:09:36,733 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:09:36,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:09:36,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:09:36,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:09:36,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:09:36,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21359.62 MB 2025-02-15 12:09:36,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29798.64 MB 2025-02-15 12:09:36,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:09:36,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33525.07 MB 2025-02-15 12:09:36,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44015.03 MB 2025-02-15 12:09:36,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:09:36,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29798.64 MB 2025-02-15 12:09:36,898 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:09:36,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:36,899 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:09:36,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:36,900 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:09:36,905 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:09:36,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:09:36,906 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:09:36,906 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:10:24,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:10:24,902 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:10:24,907 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:10:24,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:10:24,910 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:10:24,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:10:24,911 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:10:57,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:10:57,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:10:57,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.46 seconds 2025-02-15 12:10:57,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:57,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27594.87 MB 2025-02-15 12:10:57,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35023.12 MB 2025-02-15 12:10:57,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7428.24 MB 2025-02-15 12:10:57,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56600.04 MB 2025-02-15 12:10:57,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 12:10:57,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15929.97 MB 2025-02-15 12:10:57,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43861.02 MB 2025-02-15 12:10:57,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:10:57,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:10:57,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:10:57,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:57,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35023.12 MB 2025-02-15 12:10:57,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26690.95 MB 2025-02-15 12:10:57,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8332.17 MB 2025-02-15 12:10:57,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 12:10:57,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56459.53 MB 2025-02-15 12:10:57,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15789.46 MB 2025-02-15 12:10:57,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55567.72 MB 2025-02-15 12:10:59,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:10:59,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:10:59,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 12:10:59,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26690.95 MB 2025-02-15 12:10:59,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27221.79 MB 2025-02-15 12:10:59,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:10:59,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56459.53 MB 2025-02-15 12:10:59,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31178.36 MB 2025-02-15 12:10:59,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25281.17 MB 2025-02-15 12:10:59,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31200.49 MB 2025-02-15 12:10:59,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:10:59,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:10:59,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:10:59,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27221.79 MB 2025-02-15 12:10:59,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29111.32 MB 2025-02-15 12:10:59,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:10:59,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31178.36 MB 2025-02-15 12:10:59,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32122.08 MB 2025-02-15 12:10:59,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:10:59,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30528.75 MB 2025-02-15 12:10:59,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:10:59,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:10:59,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:10:59,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29111.32 MB 2025-02-15 12:10:59,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31353.18 MB 2025-02-15 12:10:59,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:10:59,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32122.08 MB 2025-02-15 12:10:59,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38728.11 MB 2025-02-15 12:10:59,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:10:59,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36897.46 MB 2025-02-15 12:10:59,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:10:59,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:10:59,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:10:59,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27221.79 MB 2025-02-15 12:10:59,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31353.18 MB 2025-02-15 12:10:59,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:10:59,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31178.36 MB 2025-02-15 12:10:59,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38728.11 MB 2025-02-15 12:10:59,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:10:59,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36897.46 MB 2025-02-15 12:10:59,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:10:59,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:10:59,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:10:59,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32886.72 MB 2025-02-15 12:10:59,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33653.72 MB 2025-02-15 12:10:59,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:10:59,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38728.11 MB 2025-02-15 12:10:59,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39143.34 MB 2025-02-15 12:10:59,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:10:59,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34361.51 MB 2025-02-15 12:10:59,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:10:59,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:10:59,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:10:59,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34066.61 MB 2025-02-15 12:10:59,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34295.78 MB 2025-02-15 12:10:59,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-15 12:10:59,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39143.34 MB 2025-02-15 12:10:59,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39143.34 MB 2025-02-15 12:10:59,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:10:59,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34515.09 MB 2025-02-15 12:10:59,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:10:59,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:10:59,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.99 seconds 2025-02-15 12:10:59,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:10:59,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20281.79 MB 2025-02-15 12:10:59,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34496.85 MB 2025-02-15 12:10:59,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14215.06 MB 2025-02-15 12:10:59,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56600.04 MB 2025-02-15 12:10:59,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39143.34 MB 2025-02-15 12:10:59,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17456.69 MB 2025-02-15 12:10:59,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34515.09 MB 2025-02-15 12:11:00,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:11:00,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:11:00,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:11:00,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:11:00,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34496.85 MB 2025-02-15 12:11:00,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.18 MB 2025-02-15 12:11:00,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9210.67 MB 2025-02-15 12:11:00,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39143.34 MB 2025-02-15 12:11:00,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39143.34 MB 2025-02-15 12:11:00,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:11:00,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37008.52 MB 2025-02-15 12:11:00,193 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:11:00,194 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:11:00,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:11:00,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:11:00,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:11:00,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:11:00,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.18 MB 2025-02-15 12:11:00,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33725.20 MB 2025-02-15 12:11:00,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:11:00,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39143.34 MB 2025-02-15 12:11:00,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47534.05 MB 2025-02-15 12:11:00,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:11:00,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33725.20 MB 2025-02-15 12:11:00,359 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:11:00,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:11:00,361 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:11:00,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:11:00,362 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:11:00,366 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:11:00,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:11:00,367 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:11:00,368 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:12:27,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:27,013 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:12:27,021 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:12:27,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:27,028 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 907, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:12:27,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:27,030 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 907, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:12:41,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:12:41,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:12:41,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.08 seconds 2025-02-15 12:12:41,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:41,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.83 MB 2025-02-15 12:12:41,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22499.57 MB 2025-02-15 12:12:41,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3210.74 MB 2025-02-15 12:12:41,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60119.06 MB 2025-02-15 12:12:41,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28070.38 MB 2025-02-15 12:12:41,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32048.68 MB 2025-02-15 12:12:41,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31478.91 MB 2025-02-15 12:12:41,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:12:41,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:12:41,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:12:41,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:41,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22499.57 MB 2025-02-15 12:12:41,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20493.06 MB 2025-02-15 12:12:41,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2006.51 MB 2025-02-15 12:12:41,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28070.38 MB 2025-02-15 12:12:41,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35488.01 MB 2025-02-15 12:12:41,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7417.63 MB 2025-02-15 12:12:41,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32722.93 MB 2025-02-15 12:12:43,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:12:43,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:12:43,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:12:43,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20493.06 MB 2025-02-15 12:12:43,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21023.90 MB 2025-02-15 12:12:43,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:12:43,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35488.01 MB 2025-02-15 12:12:43,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-15 12:12:43,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9212.79 MB 2025-02-15 12:12:43,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25002.45 MB 2025-02-15 12:12:43,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:12:43,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:12:43,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:12:43,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21023.90 MB 2025-02-15 12:12:43,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22913.44 MB 2025-02-15 12:12:43,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:12:43,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-15 12:12:43,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27218.94 MB 2025-02-15 12:12:43,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:12:43,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24330.87 MB 2025-02-15 12:12:43,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:12:43,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:12:43,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:12:43,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22913.44 MB 2025-02-15 12:12:43,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25155.29 MB 2025-02-15 12:12:43,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:12:43,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27218.94 MB 2025-02-15 12:12:43,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32881.25 MB 2025-02-15 12:12:43,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:12:43,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30699.57 MB 2025-02-15 12:12:43,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:12:43,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:12:43,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:12:43,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21023.90 MB 2025-02-15 12:12:43,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25155.29 MB 2025-02-15 12:12:43,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:12:43,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-15 12:12:43,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32881.25 MB 2025-02-15 12:12:43,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:12:43,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30699.57 MB 2025-02-15 12:12:43,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:12:43,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:12:43,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:12:43,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26688.84 MB 2025-02-15 12:12:43,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27455.84 MB 2025-02-15 12:12:43,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:12:43,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32881.25 MB 2025-02-15 12:12:43,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-15 12:12:43,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:12:43,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28163.63 MB 2025-02-15 12:12:43,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:12:43,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:12:43,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:12:43,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27868.73 MB 2025-02-15 12:12:43,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28098.68 MB 2025-02-15 12:12:43,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.96 MB 2025-02-15 12:12:43,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33296.48 MB 2025-02-15 12:12:43,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-15 12:12:43,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:12:43,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28313.02 MB 2025-02-15 12:12:43,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:12:43,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:12:43,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.48 seconds 2025-02-15 12:12:43,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16128.77 MB 2025-02-15 12:12:43,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28299.76 MB 2025-02-15 12:12:43,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12170.99 MB 2025-02-15 12:12:43,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60119.06 MB 2025-02-15 12:12:43,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-15 12:12:43,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26822.57 MB 2025-02-15 12:12:43,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28313.02 MB 2025-02-15 12:12:43,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:12:43,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:12:43,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:12:43,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28299.76 MB 2025-02-15 12:12:43,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21133.16 MB 2025-02-15 12:12:43,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7166.60 MB 2025-02-15 12:12:43,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33296.48 MB 2025-02-15 12:12:43,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-15 12:12:43,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:12:43,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30811.42 MB 2025-02-15 12:12:43,803 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:12:43,803 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:12:43,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:12:43,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:12:43,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:12:43,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:12:43,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21133.16 MB 2025-02-15 12:12:43,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29572.18 MB 2025-02-15 12:12:43,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:12:43,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33296.48 MB 2025-02-15 12:12:43,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41687.19 MB 2025-02-15 12:12:43,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:12:43,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29572.18 MB 2025-02-15 12:12:43,973 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:12:43,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:43,974 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:12:43,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:43,975 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:12:43,980 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:12:43,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:12:43,981 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:12:43,981 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:13:41,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:13:41,828 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:13:41,833 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:13:41,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:13:41,838 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:13:41,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:13:41,839 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:14:12,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:14:12,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:14:12,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.73 seconds 2025-02-15 12:14:12,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:12,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.47 MB 2025-02-15 12:14:12,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33835.81 MB 2025-02-15 12:14:12,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7028.34 MB 2025-02-15 12:14:12,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54272.20 MB 2025-02-15 12:14:12,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40277.90 MB 2025-02-15 12:14:12,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13994.30 MB 2025-02-15 12:14:12,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42847.12 MB 2025-02-15 12:14:12,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:14:12,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:14:12,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:14:12,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:12,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33835.81 MB 2025-02-15 12:14:12,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.45 MB 2025-02-15 12:14:12,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7733.37 MB 2025-02-15 12:14:12,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40277.90 MB 2025-02-15 12:14:12,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54809.07 MB 2025-02-15 12:14:12,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14531.17 MB 2025-02-15 12:14:12,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53558.09 MB 2025-02-15 12:14:14,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:14:14,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:14:14,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:14:14,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:14,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.45 MB 2025-02-15 12:14:14,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26633.29 MB 2025-02-15 12:14:14,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:14:14,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54809.07 MB 2025-02-15 12:14:14,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30477.91 MB 2025-02-15 12:14:14,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24331.16 MB 2025-02-15 12:14:14,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30612.87 MB 2025-02-15 12:14:14,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:14:14,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:14:14,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:14:14,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:14,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-15 12:14:14,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28522.82 MB 2025-02-15 12:14:14,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:14:14,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-15 12:14:14,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-15 12:14:14,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:14:14,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.25 MB 2025-02-15 12:14:14,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:14:14,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:14:14,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:14:14,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:14,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28522.82 MB 2025-02-15 12:14:14,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-15 12:14:14,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:14:14,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-15 12:14:14,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-15 12:14:14,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:14:14,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-15 12:14:14,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:14:14,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:14:14,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:14:14,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:14,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-15 12:14:14,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-15 12:14:14,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:14:14,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-15 12:14:14,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-15 12:14:14,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:14:14,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-15 12:14:15,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:14:15,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:14:15,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:14:15,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:15,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32298.22 MB 2025-02-15 12:14:15,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33065.22 MB 2025-02-15 12:14:15,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:14:15,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-15 12:14:15,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:14:15,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:14:15,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.01 MB 2025-02-15 12:14:15,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:14:15,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:14:15,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:14:15,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:15,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33478.11 MB 2025-02-15 12:14:15,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33707.02 MB 2025-02-15 12:14:15,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 12:14:15,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38442.89 MB 2025-02-15 12:14:15,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:14:15,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:14:15,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.16 MB 2025-02-15 12:14:15,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:14:15,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:14:15,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.25 seconds 2025-02-15 12:14:15,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:15,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19888.09 MB 2025-02-15 12:14:15,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33907.87 MB 2025-02-15 12:14:15,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14019.78 MB 2025-02-15 12:14:15,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54272.20 MB 2025-02-15 12:14:15,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:14:15,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15829.30 MB 2025-02-15 12:14:15,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.16 MB 2025-02-15 12:14:15,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:14:15,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:14:15,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:14:15,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:15,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33907.87 MB 2025-02-15 12:14:15,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24888.69 MB 2025-02-15 12:14:15,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9019.18 MB 2025-02-15 12:14:15,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38442.89 MB 2025-02-15 12:14:15,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:14:15,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:14:15,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36416.47 MB 2025-02-15 12:14:15,380 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 12:14:15,380 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:14:15,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:14:15,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:14:15,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:14:15,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:14:15,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24888.69 MB 2025-02-15 12:14:15,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33317.82 MB 2025-02-15 12:14:15,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 12:14:15,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38442.89 MB 2025-02-15 12:14:15,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42633.00 MB 2025-02-15 12:14:15,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 12:14:15,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33317.82 MB 2025-02-15 12:14:15,554 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 12:14:15,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:14:15,556 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:14:15,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:14:15,557 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:14:15,561 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:14:15,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:14:15,563 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:14:15,563 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:15:10,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:10,947 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:15:10,952 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:15:10,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:10,955 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1215, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:15:10,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:10,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1215, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:15:29,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:15:29,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:15:29,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.80 seconds 2025-02-15 12:15:29,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:29,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21435.02 MB 2025-02-15 12:15:29,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25734.84 MB 2025-02-15 12:15:29,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4299.82 MB 2025-02-15 12:15:29,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51013.22 MB 2025-02-15 12:15:29,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33351.01 MB 2025-02-15 12:15:29,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17662.21 MB 2025-02-15 12:15:29,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34613.00 MB 2025-02-15 12:15:29,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:15:29,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:15:29,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:15:29,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:29,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25734.84 MB 2025-02-15 12:15:29,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22094.26 MB 2025-02-15 12:15:29,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3640.58 MB 2025-02-15 12:15:29,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33351.01 MB 2025-02-15 12:15:29,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41888.51 MB 2025-02-15 12:15:29,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8537.51 MB 2025-02-15 12:15:29,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38567.67 MB 2025-02-15 12:15:31,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:15:31,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:15:31,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:15:31,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:31,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22094.26 MB 2025-02-15 12:15:31,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22625.10 MB 2025-02-15 12:15:31,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:15:31,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41888.51 MB 2025-02-15 12:15:31,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24859.64 MB 2025-02-15 12:15:31,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17028.87 MB 2025-02-15 12:15:31,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26604.68 MB 2025-02-15 12:15:31,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:15:31,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:15:31,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:15:31,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:31,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.10 MB 2025-02-15 12:15:31,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24514.63 MB 2025-02-15 12:15:31,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:15:31,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24859.64 MB 2025-02-15 12:15:31,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27690.80 MB 2025-02-15 12:15:31,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:15:31,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25932.06 MB 2025-02-15 12:15:32,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:15:32,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:15:32,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:15:32,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24514.63 MB 2025-02-15 12:15:32,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26756.49 MB 2025-02-15 12:15:32,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:15:32,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27690.80 MB 2025-02-15 12:15:32,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34296.82 MB 2025-02-15 12:15:32,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:15:32,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32300.77 MB 2025-02-15 12:15:32,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:15:32,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:15:32,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:15:32,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.10 MB 2025-02-15 12:15:32,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26756.49 MB 2025-02-15 12:15:32,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:15:32,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24859.64 MB 2025-02-15 12:15:32,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34296.82 MB 2025-02-15 12:15:32,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-15 12:15:32,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32300.77 MB 2025-02-15 12:15:32,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:15:32,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:15:32,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:15:32,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28290.03 MB 2025-02-15 12:15:32,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29057.03 MB 2025-02-15 12:15:32,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:15:32,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34296.82 MB 2025-02-15 12:15:32,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 12:15:32,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:15:32,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29764.82 MB 2025-02-15 12:15:32,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:15:32,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:15:32,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:15:32,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29469.92 MB 2025-02-15 12:15:32,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29697.52 MB 2025-02-15 12:15:32,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.60 MB 2025-02-15 12:15:32,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-15 12:15:32,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 12:15:32,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:15:32,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29932.38 MB 2025-02-15 12:15:32,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:15:32,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:15:32,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.24 seconds 2025-02-15 12:15:32,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17201.86 MB 2025-02-15 12:15:32,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29898.39 MB 2025-02-15 12:15:32,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12696.53 MB 2025-02-15 12:15:32,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51013.22 MB 2025-02-15 12:15:32,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 12:15:32,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16301.16 MB 2025-02-15 12:15:32,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29932.38 MB 2025-02-15 12:15:32,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:15:32,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:15:32,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:15:32,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29898.39 MB 2025-02-15 12:15:32,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22203.21 MB 2025-02-15 12:15:32,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7695.19 MB 2025-02-15 12:15:32,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-15 12:15:32,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-15 12:15:32,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:15:32,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32407.60 MB 2025-02-15 12:15:32,483 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 12:15:32,483 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:15:32,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:15:32,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:15:32,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:15:32,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:15:32,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22203.21 MB 2025-02-15 12:15:32,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30633.88 MB 2025-02-15 12:15:32,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-15 12:15:32,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-15 12:15:32,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43094.38 MB 2025-02-15 12:15:32,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 12:15:32,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30633.88 MB 2025-02-15 12:15:32,648 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 12:15:32,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:32,649 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:15:32,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:32,650 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:15:32,655 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:15:32,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:32,656 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:15:32,656 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:15:42,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:42,522 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:15:42,527 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:15:42,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:42,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1146, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:15:42,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:15:42,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1146, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:16:00,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:16:00,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:16:00,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.86 seconds 2025-02-15 12:16:00,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:00,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20954.22 MB 2025-02-15 12:16:00,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25010.11 MB 2025-02-15 12:16:00,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4055.89 MB 2025-02-15 12:16:00,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55666.80 MB 2025-02-15 12:16:00,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28915.53 MB 2025-02-15 12:16:00,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26751.27 MB 2025-02-15 12:16:00,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33823.54 MB 2025-02-15 12:16:00,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:16:00,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:16:00,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:16:00,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:00,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25010.11 MB 2025-02-15 12:16:00,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21736.60 MB 2025-02-15 12:16:00,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3273.51 MB 2025-02-15 12:16:00,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28915.53 MB 2025-02-15 12:16:00,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39036.39 MB 2025-02-15 12:16:00,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10120.86 MB 2025-02-15 12:16:00,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37228.15 MB 2025-02-15 12:16:02,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:16:02,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:16:02,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:16:02,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21736.60 MB 2025-02-15 12:16:02,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22267.44 MB 2025-02-15 12:16:02,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:16:02,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39036.39 MB 2025-02-15 12:16:02,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26984.05 MB 2025-02-15 12:16:02,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12052.33 MB 2025-02-15 12:16:02,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26246.35 MB 2025-02-15 12:16:02,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:16:02,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:16:02,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:16:02,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22267.44 MB 2025-02-15 12:16:02,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24156.97 MB 2025-02-15 12:16:02,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:16:02,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26984.05 MB 2025-02-15 12:16:02,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27927.77 MB 2025-02-15 12:16:02,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:16:02,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25574.40 MB 2025-02-15 12:16:02,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:16:02,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:16:02,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:16:02,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24156.97 MB 2025-02-15 12:16:02,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26398.83 MB 2025-02-15 12:16:02,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:16:02,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27927.77 MB 2025-02-15 12:16:02,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33590.08 MB 2025-02-15 12:16:02,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:16:02,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31943.11 MB 2025-02-15 12:16:02,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:16:02,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:16:02,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:16:02,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22267.44 MB 2025-02-15 12:16:02,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26398.83 MB 2025-02-15 12:16:02,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:16:02,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26984.05 MB 2025-02-15 12:16:02,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33590.08 MB 2025-02-15 12:16:02,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:16:02,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31943.11 MB 2025-02-15 12:16:02,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:16:02,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:16:02,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:16:02,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27932.37 MB 2025-02-15 12:16:02,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28699.37 MB 2025-02-15 12:16:02,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:16:02,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33590.08 MB 2025-02-15 12:16:02,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 12:16:02,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:16:02,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29407.16 MB 2025-02-15 12:16:02,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:16:02,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:16:02,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:16:02,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29112.26 MB 2025-02-15 12:16:02,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29340.16 MB 2025-02-15 12:16:02,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.90 MB 2025-02-15 12:16:02,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 12:16:02,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 12:16:02,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:16:02,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29576.12 MB 2025-02-15 12:16:02,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:16:02,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:16:02,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.31 seconds 2025-02-15 12:16:02,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:02,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16961.46 MB 2025-02-15 12:16:02,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29541.02 MB 2025-02-15 12:16:02,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12579.55 MB 2025-02-15 12:16:02,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55666.80 MB 2025-02-15 12:16:02,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 12:16:02,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21661.48 MB 2025-02-15 12:16:02,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29576.12 MB 2025-02-15 12:16:03,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:16:03,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:16:03,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:16:03,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:03,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29541.02 MB 2025-02-15 12:16:03,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21957.79 MB 2025-02-15 12:16:03,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7583.23 MB 2025-02-15 12:16:03,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 12:16:03,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34005.32 MB 2025-02-15 12:16:03,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:16:03,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32045.92 MB 2025-02-15 12:16:03,135 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 12:16:03,136 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:16:03,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:16:03,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:16:03,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:16:03,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:16:03,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21957.79 MB 2025-02-15 12:16:03,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30374.39 MB 2025-02-15 12:16:03,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 12:16:03,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34005.32 MB 2025-02-15 12:16:03,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42372.96 MB 2025-02-15 12:16:03,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 12:16:03,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30374.39 MB 2025-02-15 12:16:03,299 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 12:16:03,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:16:03,300 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:16:03,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:16:03,301 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:16:03,306 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:16:03,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:16:03,307 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:16:03,307 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:17:37,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:37,240 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:17:37,248 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:17:37,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:37,254 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:17:37,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:37,256 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:17:40,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:17:40,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:17:40,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-15 12:17:40,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:40,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-15 12:17:40,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14828.46 MB 2025-02-15 12:17:40,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-15 12:17:40,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50740.59 MB 2025-02-15 12:17:40,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19623.05 MB 2025-02-15 12:17:40,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31117.54 MB 2025-02-15 12:17:40,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23673.44 MB 2025-02-15 12:17:40,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:17:40,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:17:40,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:17:40,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:40,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14828.46 MB 2025-02-15 12:17:40,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15131.95 MB 2025-02-15 12:17:40,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 303.49 MB 2025-02-15 12:17:40,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19623.05 MB 2025-02-15 12:17:40,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19623.05 MB 2025-02-15 12:17:40,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:17:40,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17314.68 MB 2025-02-15 12:17:40,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:17:40,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:17:40,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.87 seconds 2025-02-15 12:17:40,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:40,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15131.95 MB 2025-02-15 12:17:40,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15366.85 MB 2025-02-15 12:17:40,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 12:17:40,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19623.05 MB 2025-02-15 12:17:40,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17897.10 MB 2025-02-15 12:17:40,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1725.96 MB 2025-02-15 12:17:40,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19302.64 MB 2025-02-15 12:17:40,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:17:40,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:17:40,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:17:40,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:40,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15366.78 MB 2025-02-15 12:17:40,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16202.70 MB 2025-02-15 12:17:40,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 12:17:40,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 12:17:40,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17897.10 MB 2025-02-15 12:17:40,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:17:40,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16829.92 MB 2025-02-15 12:17:41,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:17:41,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:17:41,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:17:41,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16202.70 MB 2025-02-15 12:17:41,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17194.76 MB 2025-02-15 12:17:41,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 12:17:41,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 12:17:41,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20833.11 MB 2025-02-15 12:17:41,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-15 12:17:41,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.90 MB 2025-02-15 12:17:41,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:17:41,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:17:41,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:17:41,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15366.78 MB 2025-02-15 12:17:41,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17194.76 MB 2025-02-15 12:17:41,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 12:17:41,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17897.10 MB 2025-02-15 12:17:41,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20833.11 MB 2025-02-15 12:17:41,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-15 12:17:41,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.90 MB 2025-02-15 12:17:41,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:17:41,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:17:41,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:17:41,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17873.35 MB 2025-02-15 12:17:41,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18213.66 MB 2025-02-15 12:17:41,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 12:17:41,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20833.11 MB 2025-02-15 12:17:41,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-15 12:17:41,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-15 12:17:41,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18533.45 MB 2025-02-15 12:17:41,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:17:41,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:17:41,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:17:41,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18396.37 MB 2025-02-15 12:17:41,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18625.44 MB 2025-02-15 12:17:41,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-15 12:17:41,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21013.46 MB 2025-02-15 12:17:41,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-15 12:17:41,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:17:41,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18652.64 MB 2025-02-15 12:17:41,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:17:41,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:17:41,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.97 seconds 2025-02-15 12:17:41,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13585.39 MB 2025-02-15 12:17:41,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18826.51 MB 2025-02-15 12:17:41,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5241.12 MB 2025-02-15 12:17:41,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50740.59 MB 2025-02-15 12:17:41,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-15 12:17:41,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29727.13 MB 2025-02-15 12:17:41,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18826.51 MB 2025-02-15 12:17:41,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:17:41,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:17:41,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 12:17:41,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18826.51 MB 2025-02-15 12:17:41,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17538.03 MB 2025-02-15 12:17:41,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1288.48 MB 2025-02-15 12:17:41,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21013.46 MB 2025-02-15 12:17:41,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-15 12:17:41,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:17:41,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19061.59 MB 2025-02-15 12:17:41,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:17:41,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:17:41,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:17:41,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:17:41,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:17:41,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:17:41,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17538.03 MB 2025-02-15 12:17:41,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25977.05 MB 2025-02-15 12:17:41,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:17:41,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21013.46 MB 2025-02-15 12:17:41,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31503.42 MB 2025-02-15 12:17:41,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:17:41,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25977.05 MB 2025-02-15 12:17:41,827 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:17:41,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:41,830 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:17:41,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:41,831 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:17:41,839 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:17:41,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:17:41,841 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:17:41,841 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:18:39,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:18:39,706 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:18:39,713 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:18:39,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:18:39,720 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:18:39,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:18:39,722 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:19:07,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:19:07,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:19:07,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.81 seconds 2025-02-15 12:19:07,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:07,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25518.36 MB 2025-02-15 12:19:07,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.00 MB 2025-02-15 12:19:07,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-15 12:19:07,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44088.43 MB 2025-02-15 12:19:07,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39634.08 MB 2025-02-15 12:19:07,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4454.35 MB 2025-02-15 12:19:07,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40878.54 MB 2025-02-15 12:19:07,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:19:07,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:19:07,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:19:07,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:07,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31892.00 MB 2025-02-15 12:19:07,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25140.69 MB 2025-02-15 12:19:07,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-15 12:19:07,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39634.08 MB 2025-02-15 12:19:07,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53450.11 MB 2025-02-15 12:19:07,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13816.04 MB 2025-02-15 12:19:07,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50533.93 MB 2025-02-15 12:19:09,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:19:09,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:19:09,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:19:09,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:09,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25140.69 MB 2025-02-15 12:19:09,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.53 MB 2025-02-15 12:19:09,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:19:09,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-15 12:19:09,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 12:19:09,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18775.80 MB 2025-02-15 12:19:09,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29650.08 MB 2025-02-15 12:19:09,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:19:09,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:19:09,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:19:09,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:09,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-15 12:19:09,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27561.06 MB 2025-02-15 12:19:09,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:19:09,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 12:19:09,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-15 12:19:09,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:19:09,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28978.49 MB 2025-02-15 12:19:09,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:19:09,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:19:09,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:19:09,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:09,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27561.06 MB 2025-02-15 12:19:09,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-15 12:19:09,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:19:09,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 12:19:09,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37505.47 MB 2025-02-15 12:19:09,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:19:09,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-15 12:19:09,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:19:09,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:19:09,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:19:09,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:09,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-15 12:19:09,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-15 12:19:09,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:19:09,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-15 12:19:09,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37505.47 MB 2025-02-15 12:19:09,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:19:09,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-15 12:19:10,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:19:10,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:19:10,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:19:10,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:10,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31336.46 MB 2025-02-15 12:19:10,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32103.46 MB 2025-02-15 12:19:10,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:19:10,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37505.47 MB 2025-02-15 12:19:10,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 12:19:10,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:19:10,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32811.25 MB 2025-02-15 12:19:10,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:19:10,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:19:10,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:19:10,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:10,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32516.35 MB 2025-02-15 12:19:10,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32744.61 MB 2025-02-15 12:19:10,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-15 12:19:10,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 12:19:10,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 12:19:10,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:19:10,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32984.83 MB 2025-02-15 12:19:10,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:19:10,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:19:10,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.30 seconds 2025-02-15 12:19:10,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:10,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19243.53 MB 2025-02-15 12:19:10,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32945.54 MB 2025-02-15 12:19:10,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13702.00 MB 2025-02-15 12:19:10,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44088.43 MB 2025-02-15 12:19:10,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 12:19:10,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6167.72 MB 2025-02-15 12:19:10,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32984.83 MB 2025-02-15 12:19:10,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:19:10,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:19:10,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:19:10,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:10,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32945.54 MB 2025-02-15 12:19:10,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24245.64 MB 2025-02-15 12:19:10,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8699.90 MB 2025-02-15 12:19:10,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 12:19:10,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37920.70 MB 2025-02-15 12:19:10,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:19:10,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35455.36 MB 2025-02-15 12:19:10,310 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 12:19:10,310 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:19:10,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:19:10,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:19:10,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:19:10,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:19:10,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24245.64 MB 2025-02-15 12:19:10,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32678.14 MB 2025-02-15 12:19:10,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8432.50 MB 2025-02-15 12:19:10,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37920.70 MB 2025-02-15 12:19:10,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42112.91 MB 2025-02-15 12:19:10,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 12:19:10,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32678.14 MB 2025-02-15 12:19:10,479 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 12:19:10,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:10,481 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:19:10,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:10,482 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:19:10,486 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:19:10,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:10,487 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:19:10,487 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:19:43,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:43,812 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:19:43,817 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:19:43,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:43,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:19:43,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:19:43,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:20:01,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:20:01,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:20:01,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.67 seconds 2025-02-15 12:20:01,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:01,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20926.35 MB 2025-02-15 12:20:01,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24967.82 MB 2025-02-15 12:20:01,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-15 12:20:01,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50497.32 MB 2025-02-15 12:20:01,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28909.24 MB 2025-02-15 12:20:01,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21588.08 MB 2025-02-15 12:20:01,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33795.10 MB 2025-02-15 12:20:01,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:20:01,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:20:01,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:20:01,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:01,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24967.82 MB 2025-02-15 12:20:01,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.80 MB 2025-02-15 12:20:01,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3252.02 MB 2025-02-15 12:20:01,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28909.24 MB 2025-02-15 12:20:01,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39021.71 MB 2025-02-15 12:20:01,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10112.47 MB 2025-02-15 12:20:01,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37180.33 MB 2025-02-15 12:20:03,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:20:03,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:20:03,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:20:03,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.80 MB 2025-02-15 12:20:03,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22246.64 MB 2025-02-15 12:20:03,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:20:03,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39021.71 MB 2025-02-15 12:20:03,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26990.35 MB 2025-02-15 12:20:03,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12031.36 MB 2025-02-15 12:20:03,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26225.19 MB 2025-02-15 12:20:03,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:20:03,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:20:03,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:20:03,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22246.64 MB 2025-02-15 12:20:03,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24136.18 MB 2025-02-15 12:20:03,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:20:03,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 12:20:03,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27934.06 MB 2025-02-15 12:20:03,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:20:03,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25553.61 MB 2025-02-15 12:20:03,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:20:03,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:20:03,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:20:03,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24136.18 MB 2025-02-15 12:20:03,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26378.03 MB 2025-02-15 12:20:03,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:20:03,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27934.06 MB 2025-02-15 12:20:03,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 12:20:03,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:20:03,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31922.31 MB 2025-02-15 12:20:03,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:20:03,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:20:03,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:20:03,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22246.64 MB 2025-02-15 12:20:03,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26378.03 MB 2025-02-15 12:20:03,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:20:03,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26990.35 MB 2025-02-15 12:20:03,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-15 12:20:03,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:20:03,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31922.31 MB 2025-02-15 12:20:03,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:20:03,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:20:03,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:20:03,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27911.58 MB 2025-02-15 12:20:03,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28678.58 MB 2025-02-15 12:20:03,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:20:03,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-15 12:20:03,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34009.51 MB 2025-02-15 12:20:03,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 12:20:03,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29386.37 MB 2025-02-15 12:20:03,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:20:03,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:20:03,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:20:03,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29091.47 MB 2025-02-15 12:20:03,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29319.20 MB 2025-02-15 12:20:03,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.73 MB 2025-02-15 12:20:03,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34009.51 MB 2025-02-15 12:20:03,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34009.51 MB 2025-02-15 12:20:03,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:20:03,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29551.21 MB 2025-02-15 12:20:03,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:20:03,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:20:03,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.11 seconds 2025-02-15 12:20:03,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:03,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16947.53 MB 2025-02-15 12:20:03,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29520.05 MB 2025-02-15 12:20:03,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12572.52 MB 2025-02-15 12:20:03,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50497.32 MB 2025-02-15 12:20:03,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34009.51 MB 2025-02-15 12:20:03,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16487.81 MB 2025-02-15 12:20:03,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29551.21 MB 2025-02-15 12:20:04,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:20:04,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:20:04,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:20:04,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:04,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29520.05 MB 2025-02-15 12:20:04,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21946.70 MB 2025-02-15 12:20:04,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7573.34 MB 2025-02-15 12:20:04,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34009.51 MB 2025-02-15 12:20:04,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34009.51 MB 2025-02-15 12:20:04,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:20:04,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32027.42 MB 2025-02-15 12:20:04,220 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 12:20:04,220 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:20:04,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:20:04,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:20:04,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:20:04,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:20:04,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21946.70 MB 2025-02-15 12:20:04,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30371.66 MB 2025-02-15 12:20:04,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 12:20:04,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34009.51 MB 2025-02-15 12:20:04,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38197.53 MB 2025-02-15 12:20:04,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 12:20:04,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30371.66 MB 2025-02-15 12:20:04,388 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 12:20:04,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:20:04,390 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:20:04,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:20:04,391 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:20:04,395 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:20:04,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:20:04,396 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:20:04,396 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:21:29,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:29,685 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:21:29,690 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:21:29,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:29,694 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:21:29,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:29,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:21:42,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:21:42,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:21:42,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.41 seconds 2025-02-15 12:21:42,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18585.04 MB 2025-02-15 12:21:42,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21437.43 MB 2025-02-15 12:21:42,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-15 12:21:42,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46573.55 MB 2025-02-15 12:21:42,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24939.33 MB 2025-02-15 12:21:42,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21634.22 MB 2025-02-15 12:21:42,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30322.15 MB 2025-02-15 12:21:42,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:21:42,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:21:42,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:21:42,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21437.43 MB 2025-02-15 12:21:42,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17587.19 MB 2025-02-15 12:21:42,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3850.24 MB 2025-02-15 12:21:42,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24939.33 MB 2025-02-15 12:21:42,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24939.33 MB 2025-02-15 12:21:42,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22294.24 MB 2025-02-15 12:21:42,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:21:42,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:21:42,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:21:42,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17587.19 MB 2025-02-15 12:21:42,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17668.14 MB 2025-02-15 12:21:42,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-15 12:21:42,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24939.33 MB 2025-02-15 12:21:42,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 12:21:42,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2854.22 MB 2025-02-15 12:21:42,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21480.86 MB 2025-02-15 12:21:42,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:21:42,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:21:42,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:21:42,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.14 MB 2025-02-15 12:21:42,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17956.23 MB 2025-02-15 12:21:42,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-15 12:21:42,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22085.11 MB 2025-02-15 12:21:42,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 12:21:42,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18172.39 MB 2025-02-15 12:21:42,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:21:42,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:21:42,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:21:42,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17956.23 MB 2025-02-15 12:21:42,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.16 MB 2025-02-15 12:21:42,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-15 12:21:42,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22085.11 MB 2025-02-15 12:21:42,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 12:21:42,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19143.61 MB 2025-02-15 12:21:42,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:21:42,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:21:42,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:21:42,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.14 MB 2025-02-15 12:21:42,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.16 MB 2025-02-15 12:21:42,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.02 MB 2025-02-15 12:21:42,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22085.11 MB 2025-02-15 12:21:42,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22085.11 MB 2025-02-15 12:21:42,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19143.61 MB 2025-02-15 12:21:42,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:21:42,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:21:42,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:21:42,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18644.87 MB 2025-02-15 12:21:42,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18791.82 MB 2025-02-15 12:21:42,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-15 12:21:42,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22085.11 MB 2025-02-15 12:21:42,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22173.19 MB 2025-02-15 12:21:42,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 88.08 MB 2025-02-15 12:21:42,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18899.76 MB 2025-02-15 12:21:42,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:21:42,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:21:42,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:21:42,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18884.78 MB 2025-02-15 12:21:42,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19031.88 MB 2025-02-15 12:21:42,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.10 MB 2025-02-15 12:21:42,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22173.19 MB 2025-02-15 12:21:42,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22173.19 MB 2025-02-15 12:21:42,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19031.88 MB 2025-02-15 12:21:42,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:21:42,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:21:42,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.85 seconds 2025-02-15 12:21:42,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.88 MB 2025-02-15 12:21:42,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19163.72 MB 2025-02-15 12:21:42,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3386.85 MB 2025-02-15 12:21:42,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46573.55 MB 2025-02-15 12:21:42,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22173.19 MB 2025-02-15 12:21:42,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24400.36 MB 2025-02-15 12:21:42,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19163.72 MB 2025-02-15 12:21:42,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:21:42,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:21:42,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:21:42,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16132.74 MB 2025-02-15 12:21:42,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18109.04 MB 2025-02-15 12:21:42,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1976.30 MB 2025-02-15 12:21:42,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22173.19 MB 2025-02-15 12:21:42,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22173.19 MB 2025-02-15 12:21:42,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:21:42,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18306.65 MB 2025-02-15 12:21:42,726 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-15 12:21:42,726 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:21:42,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:21:42,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:21:42,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:21:42,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:21:42,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18109.04 MB 2025-02-15 12:21:42,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23642.31 MB 2025-02-15 12:21:42,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.26 MB 2025-02-15 12:21:42,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22173.19 MB 2025-02-15 12:21:42,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24924.65 MB 2025-02-15 12:21:42,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2751.46 MB 2025-02-15 12:21:42,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23642.31 MB 2025-02-15 12:21:42,837 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-15 12:21:42,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:42,838 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:21:42,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:42,839 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:21:42,844 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:21:42,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:21:42,845 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:21:42,846 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:23:03,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:03,589 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:23:03,596 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:23:03,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:03,602 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:23:03,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:03,604 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:23:33,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:23:33,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:23:33,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.30 seconds 2025-02-15 12:23:33,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:33,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.82 MB 2025-02-15 12:23:33,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33564.20 MB 2025-02-15 12:23:33,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-15 12:23:33,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40619.74 MB 2025-02-15 12:23:33,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37796.97 MB 2025-02-15 12:23:33,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2822.77 MB 2025-02-15 12:23:33,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42439.98 MB 2025-02-15 12:23:34,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:23:34,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:23:34,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:23:34,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:34,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33564.20 MB 2025-02-15 12:23:34,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25968.85 MB 2025-02-15 12:23:34,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7595.35 MB 2025-02-15 12:23:34,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37796.97 MB 2025-02-15 12:23:34,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52860.81 MB 2025-02-15 12:23:34,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15063.84 MB 2025-02-15 12:23:34,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53021.66 MB 2025-02-15 12:23:36,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:23:36,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:23:36,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:23:36,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25968.85 MB 2025-02-15 12:23:36,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26499.69 MB 2025-02-15 12:23:36,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:23:36,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52860.81 MB 2025-02-15 12:23:36,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28795.99 MB 2025-02-15 12:23:36,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24064.82 MB 2025-02-15 12:23:36,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30478.24 MB 2025-02-15 12:23:36,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:23:36,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:23:36,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:23:36,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26499.69 MB 2025-02-15 12:23:36,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28389.23 MB 2025-02-15 12:23:36,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:23:36,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28795.99 MB 2025-02-15 12:23:36,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31627.15 MB 2025-02-15 12:23:36,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:23:36,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29806.66 MB 2025-02-15 12:23:36,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:23:36,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:23:36,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:23:36,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28389.23 MB 2025-02-15 12:23:36,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30631.08 MB 2025-02-15 12:23:36,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:23:36,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31627.15 MB 2025-02-15 12:23:36,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38233.18 MB 2025-02-15 12:23:36,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:23:36,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36175.37 MB 2025-02-15 12:23:36,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:23:36,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:23:36,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:23:36,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26499.69 MB 2025-02-15 12:23:36,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30631.08 MB 2025-02-15 12:23:36,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:23:36,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28795.99 MB 2025-02-15 12:23:36,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38233.18 MB 2025-02-15 12:23:36,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-15 12:23:36,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36175.37 MB 2025-02-15 12:23:36,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:23:36,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:23:36,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:23:36,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32164.63 MB 2025-02-15 12:23:36,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32931.63 MB 2025-02-15 12:23:36,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:23:36,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38233.18 MB 2025-02-15 12:23:36,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38648.41 MB 2025-02-15 12:23:36,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:23:36,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33639.42 MB 2025-02-15 12:23:36,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:23:36,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:23:36,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:23:36,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33344.52 MB 2025-02-15 12:23:36,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33573.40 MB 2025-02-15 12:23:36,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-15 12:23:36,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38648.41 MB 2025-02-15 12:23:36,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38648.41 MB 2025-02-15 12:23:36,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:23:36,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33757.60 MB 2025-02-15 12:23:36,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:23:36,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:23:36,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.83 seconds 2025-02-15 12:23:36,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19798.03 MB 2025-02-15 12:23:36,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33774.21 MB 2025-02-15 12:23:36,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13976.18 MB 2025-02-15 12:23:36,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37258.00 MB 2025-02-15 12:23:36,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38648.41 MB 2025-02-15 12:23:36,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1390.41 MB 2025-02-15 12:23:36,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33774.21 MB 2025-02-15 12:23:36,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:23:36,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:23:36,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:23:36,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33774.21 MB 2025-02-15 12:23:36,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24798.28 MB 2025-02-15 12:23:36,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8975.93 MB 2025-02-15 12:23:36,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38648.41 MB 2025-02-15 12:23:36,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38648.41 MB 2025-02-15 12:23:36,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:23:36,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36282.49 MB 2025-02-15 12:23:36,730 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 12:23:36,731 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:23:36,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:23:36,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:23:36,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:23:36,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:23:36,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24798.28 MB 2025-02-15 12:23:36,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33225.61 MB 2025-02-15 12:23:36,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 12:23:36,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38648.41 MB 2025-02-15 12:23:36,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47028.63 MB 2025-02-15 12:23:36,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 12:23:36,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33225.61 MB 2025-02-15 12:23:36,909 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 12:23:36,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:36,911 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:23:36,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:36,912 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:23:36,917 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:23:36,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:36,918 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:23:36,918 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:23:50,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:50,037 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:23:50,044 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:23:50,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:50,050 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1757, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:23:50,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:23:50,051 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1757, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:24:17,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:24:17,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:24:17,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.55 seconds 2025-02-15 12:24:17,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:17,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25211.76 MB 2025-02-15 12:24:17,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31429.82 MB 2025-02-15 12:24:17,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6218.06 MB 2025-02-15 12:24:17,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55408.85 MB 2025-02-15 12:24:17,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37079.74 MB 2025-02-15 12:24:17,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18329.11 MB 2025-02-15 12:24:17,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40345.44 MB 2025-02-15 12:24:17,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:24:17,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:24:17,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:24:17,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:17,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.82 MB 2025-02-15 12:24:17,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24912.99 MB 2025-02-15 12:24:17,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6516.83 MB 2025-02-15 12:24:17,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37079.74 MB 2025-02-15 12:24:17,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50155.49 MB 2025-02-15 12:24:17,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13075.74 MB 2025-02-15 12:24:17,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47898.87 MB 2025-02-15 12:24:19,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:24:19,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:24:19,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:24:19,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:19,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.99 MB 2025-02-15 12:24:19,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25443.84 MB 2025-02-15 12:24:19,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:24:19,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50155.49 MB 2025-02-15 12:24:19,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29624.37 MB 2025-02-15 12:24:19,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20531.12 MB 2025-02-15 12:24:19,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29422.38 MB 2025-02-15 12:24:19,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:24:19,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:24:19,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:24:19,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:19,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25443.84 MB 2025-02-15 12:24:19,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27333.37 MB 2025-02-15 12:24:19,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:24:19,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29624.37 MB 2025-02-15 12:24:19,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30568.09 MB 2025-02-15 12:24:19,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:24:19,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28750.80 MB 2025-02-15 12:24:19,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:24:19,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:24:19,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:24:19,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:19,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27333.37 MB 2025-02-15 12:24:19,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29575.23 MB 2025-02-15 12:24:19,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:24:19,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30568.09 MB 2025-02-15 12:24:19,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37174.12 MB 2025-02-15 12:24:19,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:24:19,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.51 MB 2025-02-15 12:24:19,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:24:19,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:24:19,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:24:19,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:19,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25443.84 MB 2025-02-15 12:24:19,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29575.23 MB 2025-02-15 12:24:19,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:24:19,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29624.37 MB 2025-02-15 12:24:19,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37174.12 MB 2025-02-15 12:24:19,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:24:19,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.51 MB 2025-02-15 12:24:20,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:24:20,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:24:20,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:24:20,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:20,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31108.77 MB 2025-02-15 12:24:20,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31875.77 MB 2025-02-15 12:24:20,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:24:20,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37174.12 MB 2025-02-15 12:24:20,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37589.35 MB 2025-02-15 12:24:20,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:24:20,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32583.56 MB 2025-02-15 12:24:20,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:24:20,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:24:20,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:24:20,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:20,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32288.66 MB 2025-02-15 12:24:20,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32517.82 MB 2025-02-15 12:24:20,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-15 12:24:20,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37589.35 MB 2025-02-15 12:24:20,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37589.35 MB 2025-02-15 12:24:20,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:24:20,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32731.84 MB 2025-02-15 12:24:20,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:24:20,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:24:20,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.03 seconds 2025-02-15 12:24:20,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:20,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19090.23 MB 2025-02-15 12:24:20,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32718.89 MB 2025-02-15 12:24:20,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13628.65 MB 2025-02-15 12:24:20,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55408.85 MB 2025-02-15 12:24:20,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37589.35 MB 2025-02-15 12:24:20,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17819.50 MB 2025-02-15 12:24:20,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32731.84 MB 2025-02-15 12:24:20,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:24:20,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:24:20,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:24:20,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:20,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32718.89 MB 2025-02-15 12:24:20,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24094.62 MB 2025-02-15 12:24:20,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8624.27 MB 2025-02-15 12:24:20,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37589.35 MB 2025-02-15 12:24:20,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37589.35 MB 2025-02-15 12:24:20,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:24:20,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35230.56 MB 2025-02-15 12:24:20,377 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:24:20,377 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:24:20,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:24:20,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:24:20,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:24:20,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:24:20,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24094.62 MB 2025-02-15 12:24:20,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32533.65 MB 2025-02-15 12:24:20,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:24:20,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37589.35 MB 2025-02-15 12:24:20,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45980.06 MB 2025-02-15 12:24:20,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:24:20,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32533.65 MB 2025-02-15 12:24:20,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:24:20,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:24:20,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:24:20,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:24:20,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:24:20,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:24:20,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:24:20,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:24:20,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:25:28,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:28,880 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:25:28,885 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:25:28,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:28,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:25:28,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:28,890 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:25:33,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:25:33,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:25:33,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.38 seconds 2025-02-15 12:25:33,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-15 12:25:33,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15943.13 MB 2025-02-15 12:25:33,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1002.44 MB 2025-02-15 12:25:33,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58565.07 MB 2025-02-15 12:25:33,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19134.41 MB 2025-02-15 12:25:33,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39430.65 MB 2025-02-15 12:25:33,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24865.86 MB 2025-02-15 12:25:33,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:25:33,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:25:33,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:25:33,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15943.13 MB 2025-02-15 12:25:33,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14580.40 MB 2025-02-15 12:25:33,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1362.74 MB 2025-02-15 12:25:33,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19134.41 MB 2025-02-15 12:25:33,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19134.41 MB 2025-02-15 12:25:33,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:25:33,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16224.12 MB 2025-02-15 12:25:33,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:25:33,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:25:33,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:25:33,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14580.40 MB 2025-02-15 12:25:33,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14606.94 MB 2025-02-15 12:25:33,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.54 MB 2025-02-15 12:25:33,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19134.41 MB 2025-02-15 12:25:33,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18131.98 MB 2025-02-15 12:25:33,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1002.44 MB 2025-02-15 12:25:33,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15856.87 MB 2025-02-15 12:25:33,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:25:33,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:25:33,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:25:33,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-15 12:25:33,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14701.33 MB 2025-02-15 12:25:33,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.45 MB 2025-02-15 12:25:33,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18131.98 MB 2025-02-15 12:25:33,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18131.98 MB 2025-02-15 12:25:33,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:25:33,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14772.21 MB 2025-02-15 12:25:33,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:25:33,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:25:33,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:25:33,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14701.33 MB 2025-02-15 12:25:33,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14813.65 MB 2025-02-15 12:25:33,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 112.33 MB 2025-02-15 12:25:33,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18131.98 MB 2025-02-15 12:25:33,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18131.98 MB 2025-02-15 12:25:33,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:25:33,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15091.93 MB 2025-02-15 12:25:33,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:25:33,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:25:33,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:25:33,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-15 12:25:33,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14813.65 MB 2025-02-15 12:25:33,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.78 MB 2025-02-15 12:25:33,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18131.98 MB 2025-02-15 12:25:33,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18131.98 MB 2025-02-15 12:25:33,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:25:33,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15091.93 MB 2025-02-15 12:25:33,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:25:33,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:25:33,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:25:33,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14890.72 MB 2025-02-15 12:25:33,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14929.07 MB 2025-02-15 12:25:33,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 38.35 MB 2025-02-15 12:25:33,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18131.98 MB 2025-02-15 12:25:33,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18148.75 MB 2025-02-15 12:25:33,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16.78 MB 2025-02-15 12:25:33,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14979.74 MB 2025-02-15 12:25:33,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:25:33,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:25:33,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:25:33,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14949.73 MB 2025-02-15 12:25:33,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.22 MB 2025-02-15 12:25:33,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.49 MB 2025-02-15 12:25:33,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18148.75 MB 2025-02-15 12:25:33,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18148.75 MB 2025-02-15 12:25:33,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:25:33,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14975.22 MB 2025-02-15 12:25:33,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:25:33,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:25:33,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.54 seconds 2025-02-15 12:25:33,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13954.70 MB 2025-02-15 12:25:33,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15022.36 MB 2025-02-15 12:25:33,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1067.66 MB 2025-02-15 12:25:33,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58565.07 MB 2025-02-15 12:25:33,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18148.75 MB 2025-02-15 12:25:33,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40416.31 MB 2025-02-15 12:25:33,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15022.36 MB 2025-02-15 12:25:33,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:25:33,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:25:33,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:25:33,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15022.36 MB 2025-02-15 12:25:33,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.05 MB 2025-02-15 12:25:33,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 706.69 MB 2025-02-15 12:25:33,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18148.75 MB 2025-02-15 12:25:33,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18152.95 MB 2025-02-15 12:25:33,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:25:33,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15799.71 MB 2025-02-15 12:25:33,512 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1903, cut from 1905 2025-02-15 12:25:33,512 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:25:33,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:25:33,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:25:33,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:25:33,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:25:33,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14781.65 MB 2025-02-15 12:25:33,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16760.87 MB 2025-02-15 12:25:33,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1979.22 MB 2025-02-15 12:25:33,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18152.95 MB 2025-02-15 12:25:33,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19136.51 MB 2025-02-15 12:25:33,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 983.56 MB 2025-02-15 12:25:33,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16760.87 MB 2025-02-15 12:25:33,553 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1695] 2025-02-15 12:25:33,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:33,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:25:33,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:33,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:25:33,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:25:33,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:33,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:25:33,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:25:52,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:52,779 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:25:52,787 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:25:52,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:52,793 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:25:52,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:25:52,795 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:26:13,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:26:13,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:26:13,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.58 seconds 2025-02-15 12:26:13,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:13,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22217.15 MB 2025-02-15 12:26:13,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26913.33 MB 2025-02-15 12:26:13,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-15 12:26:13,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30352.08 MB 2025-02-15 12:26:13,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30425.48 MB 2025-02-15 12:26:13,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 73.40 MB 2025-02-15 12:26:13,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35765.38 MB 2025-02-15 12:26:13,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:26:13,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:26:13,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:26:13,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:13,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26913.33 MB 2025-02-15 12:26:13,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22678.53 MB 2025-02-15 12:26:13,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4234.80 MB 2025-02-15 12:26:13,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30425.48 MB 2025-02-15 12:26:13,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41339.06 MB 2025-02-15 12:26:13,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10913.58 MB 2025-02-15 12:26:13,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40129.75 MB 2025-02-15 12:26:15,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:26:15,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:26:15,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:26:15,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22678.53 MB 2025-02-15 12:26:15,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23209.37 MB 2025-02-15 12:26:15,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:26:15,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41339.06 MB 2025-02-15 12:26:15,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27648.85 MB 2025-02-15 12:26:15,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13690.21 MB 2025-02-15 12:26:15,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27189.99 MB 2025-02-15 12:26:15,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:26:15,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:26:15,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:26:15,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23209.37 MB 2025-02-15 12:26:15,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25098.90 MB 2025-02-15 12:26:15,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:26:15,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27648.85 MB 2025-02-15 12:26:15,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28592.57 MB 2025-02-15 12:26:15,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:26:15,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26516.33 MB 2025-02-15 12:26:15,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:26:15,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:26:15,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:26:15,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25098.90 MB 2025-02-15 12:26:15,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27341.81 MB 2025-02-15 12:26:15,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 12:26:15,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28592.57 MB 2025-02-15 12:26:15,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34963.72 MB 2025-02-15 12:26:15,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 12:26:15,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32886.09 MB 2025-02-15 12:26:15,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:26:15,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:26:15,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:26:15,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23209.37 MB 2025-02-15 12:26:15,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27341.81 MB 2025-02-15 12:26:15,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 12:26:15,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27648.85 MB 2025-02-15 12:26:15,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34963.72 MB 2025-02-15 12:26:15,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 12:26:15,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32886.09 MB 2025-02-15 12:26:15,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:26:15,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:26:15,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:26:15,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28875.35 MB 2025-02-15 12:26:15,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29642.35 MB 2025-02-15 12:26:15,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:26:15,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34963.72 MB 2025-02-15 12:26:15,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35378.95 MB 2025-02-15 12:26:15,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:26:15,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30350.14 MB 2025-02-15 12:26:15,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:26:15,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:26:15,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:26:15,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30055.24 MB 2025-02-15 12:26:15,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30283.49 MB 2025-02-15 12:26:15,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-15 12:26:15,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35378.95 MB 2025-02-15 12:26:15,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35378.95 MB 2025-02-15 12:26:15,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:26:15,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.26 MB 2025-02-15 12:26:15,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:26:15,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:26:15,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.04 seconds 2025-02-15 12:26:15,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:15,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17592.93 MB 2025-02-15 12:26:15,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30484.34 MB 2025-02-15 12:26:15,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12891.42 MB 2025-02-15 12:26:15,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25727.86 MB 2025-02-15 12:26:15,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35378.95 MB 2025-02-15 12:26:15,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9651.09 MB 2025-02-15 12:26:15,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.34 MB 2025-02-15 12:26:16,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:26:16,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:26:16,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:26:16,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:16,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30484.34 MB 2025-02-15 12:26:16,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22583.91 MB 2025-02-15 12:26:16,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7900.43 MB 2025-02-15 12:26:16,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35378.95 MB 2025-02-15 12:26:16,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35378.95 MB 2025-02-15 12:26:16,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:26:16,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32984.64 MB 2025-02-15 12:26:16,121 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 12:26:16,121 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:26:16,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:26:16,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:26:16,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:26:16,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:26:16,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22583.91 MB 2025-02-15 12:26:16,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30984.85 MB 2025-02-15 12:26:16,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-15 12:26:16,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35378.95 MB 2025-02-15 12:26:16,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43731.91 MB 2025-02-15 12:26:16,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-15 12:26:16,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30984.85 MB 2025-02-15 12:26:16,285 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 12:26:16,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:26:16,287 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:26:16,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:26:16,288 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:26:16,292 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:26:16,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:26:16,293 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:26:16,293 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:27:07,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:07,224 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:27:07,230 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:27:07,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:07,235 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:27:07,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:07,236 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:27:13,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:27:13,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:27:13,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.91 seconds 2025-02-15 12:27:13,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:13,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-15 12:27:13,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-15 12:27:13,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-15 12:27:13,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56260.30 MB 2025-02-15 12:27:13,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19075.69 MB 2025-02-15 12:27:13,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37184.60 MB 2025-02-15 12:27:13,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-15 12:27:13,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:27:13,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:27:13,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:27:13,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:13,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-15 12:27:13,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17531.74 MB 2025-02-15 12:27:13,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 580.85 MB 2025-02-15 12:27:13,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19075.69 MB 2025-02-15 12:27:13,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23609.74 MB 2025-02-15 12:27:13,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4534.04 MB 2025-02-15 12:27:13,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22136.50 MB 2025-02-15 12:27:14,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:27:14,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:27:14,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.75 seconds 2025-02-15 12:27:14,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:14,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17531.74 MB 2025-02-15 12:27:14,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18021.45 MB 2025-02-15 12:27:14,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 489.70 MB 2025-02-15 12:27:14,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23609.74 MB 2025-02-15 12:27:14,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-15 12:27:14,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2594.18 MB 2025-02-15 12:27:14,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21957.24 MB 2025-02-15 12:27:14,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:27:14,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:27:14,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:27:14,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:14,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18021.45 MB 2025-02-15 12:27:14,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19764.44 MB 2025-02-15 12:27:14,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1743.00 MB 2025-02-15 12:27:14,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-15 12:27:14,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23632.81 MB 2025-02-15 12:27:14,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2617.25 MB 2025-02-15 12:27:14,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21072.02 MB 2025-02-15 12:27:15,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:27:15,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:27:15,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.33 seconds 2025-02-15 12:27:15,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19764.44 MB 2025-02-15 12:27:15,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21832.56 MB 2025-02-15 12:27:15,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2068.12 MB 2025-02-15 12:27:15,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23632.81 MB 2025-02-15 12:27:15,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28867.30 MB 2025-02-15 12:27:15,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5234.49 MB 2025-02-15 12:27:15,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26948.99 MB 2025-02-15 12:27:15,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:27:15,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:27:15,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-15 12:27:15,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18021.45 MB 2025-02-15 12:27:15,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21832.56 MB 2025-02-15 12:27:15,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.11 MB 2025-02-15 12:27:15,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-15 12:27:15,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28867.30 MB 2025-02-15 12:27:15,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7851.74 MB 2025-02-15 12:27:15,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26948.99 MB 2025-02-15 12:27:15,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:27:15,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:27:15,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:27:15,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23247.25 MB 2025-02-15 12:27:15,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23955.73 MB 2025-02-15 12:27:15,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 708.48 MB 2025-02-15 12:27:15,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28867.30 MB 2025-02-15 12:27:15,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29248.98 MB 2025-02-15 12:27:15,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-15 12:27:15,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24608.66 MB 2025-02-15 12:27:15,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:27:15,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:27:15,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:27:15,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24336.62 MB 2025-02-15 12:27:15,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24564.69 MB 2025-02-15 12:27:15,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.07 MB 2025-02-15 12:27:15,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29248.98 MB 2025-02-15 12:27:15,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29248.98 MB 2025-02-15 12:27:15,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:27:15,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24719.47 MB 2025-02-15 12:27:15,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:27:15,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:27:15,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.23 seconds 2025-02-15 12:27:15,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-15 12:27:15,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24765.76 MB 2025-02-15 12:27:15,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10476.59 MB 2025-02-15 12:27:15,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56260.30 MB 2025-02-15 12:27:15,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29248.98 MB 2025-02-15 12:27:15,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27011.32 MB 2025-02-15 12:27:15,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24765.76 MB 2025-02-15 12:27:15,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:27:15,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:27:15,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:27:15,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24765.76 MB 2025-02-15 12:27:15,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19147.92 MB 2025-02-15 12:27:15,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5617.84 MB 2025-02-15 12:27:15,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29248.98 MB 2025-02-15 12:27:15,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29248.98 MB 2025-02-15 12:27:15,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:27:15,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26272.76 MB 2025-02-15 12:27:15,752 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:27:15,752 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:27:15,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:27:15,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:27:15,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:27:15,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:27:15,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19147.92 MB 2025-02-15 12:27:15,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27586.94 MB 2025-02-15 12:27:15,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:27:15,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29248.98 MB 2025-02-15 12:27:15,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39738.93 MB 2025-02-15 12:27:15,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:27:15,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27586.94 MB 2025-02-15 12:27:15,929 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:27:15,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:15,930 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:27:15,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:15,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:27:15,936 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:27:15,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:27:15,937 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:27:15,937 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:28:12,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:12,871 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:28:12,876 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:28:12,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:12,879 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:28:12,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:12,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:28:29,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:28:29,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:28:29,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.95 seconds 2025-02-15 12:28:29,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:29,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20626.71 MB 2025-02-15 12:28:29,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24516.93 MB 2025-02-15 12:28:29,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3890.22 MB 2025-02-15 12:28:29,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52323.94 MB 2025-02-15 12:28:29,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26663.19 MB 2025-02-15 12:28:29,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25660.75 MB 2025-02-15 12:28:29,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.28 MB 2025-02-15 12:28:29,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:28:29,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:28:29,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:28:29,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:29,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24516.93 MB 2025-02-15 12:28:29,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21492.26 MB 2025-02-15 12:28:29,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3024.67 MB 2025-02-15 12:28:29,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26663.19 MB 2025-02-15 12:28:29,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36509.32 MB 2025-02-15 12:28:29,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9846.13 MB 2025-02-15 12:28:29,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36462.97 MB 2025-02-15 12:28:31,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:28:31,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:28:31,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:28:31,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:31,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21492.26 MB 2025-02-15 12:28:31,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22023.10 MB 2025-02-15 12:28:31,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:28:31,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36509.32 MB 2025-02-15 12:28:31,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24897.39 MB 2025-02-15 12:28:31,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11611.93 MB 2025-02-15 12:28:31,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26002.68 MB 2025-02-15 12:28:31,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:28:31,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:28:31,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:28:31,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:31,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22023.10 MB 2025-02-15 12:28:31,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23912.63 MB 2025-02-15 12:28:31,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:28:31,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 12:28:31,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27728.54 MB 2025-02-15 12:28:31,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:28:31,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25330.06 MB 2025-02-15 12:28:32,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:28:32,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:28:32,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:28:32,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23912.63 MB 2025-02-15 12:28:32,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.49 MB 2025-02-15 12:28:32,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:28:32,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27728.54 MB 2025-02-15 12:28:32,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-15 12:28:32,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:28:32,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.77 MB 2025-02-15 12:28:32,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:28:32,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:28:32,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:28:32,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22023.10 MB 2025-02-15 12:28:32,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.49 MB 2025-02-15 12:28:32,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:28:32,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24897.39 MB 2025-02-15 12:28:32,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-15 12:28:32,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 12:28:32,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.77 MB 2025-02-15 12:28:32,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:28:32,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:28:32,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:28:32,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27688.03 MB 2025-02-15 12:28:32,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28455.03 MB 2025-02-15 12:28:32,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:28:32,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33390.85 MB 2025-02-15 12:28:32,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 12:28:32,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:28:32,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29162.82 MB 2025-02-15 12:28:32,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:28:32,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:28:32,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:28:32,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28867.92 MB 2025-02-15 12:28:32,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29096.87 MB 2025-02-15 12:28:32,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.95 MB 2025-02-15 12:28:32,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 12:28:32,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 12:28:32,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:28:32,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29333.23 MB 2025-02-15 12:28:32,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:28:32,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:28:32,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.44 seconds 2025-02-15 12:28:32,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16797.71 MB 2025-02-15 12:28:32,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29296.81 MB 2025-02-15 12:28:32,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12499.10 MB 2025-02-15 12:28:32,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52323.94 MB 2025-02-15 12:28:32,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 12:28:32,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18517.85 MB 2025-02-15 12:28:32,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29333.23 MB 2025-02-15 12:28:32,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:28:32,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:28:32,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:28:32,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29296.81 MB 2025-02-15 12:28:32,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21785.49 MB 2025-02-15 12:28:32,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7511.33 MB 2025-02-15 12:28:32,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 12:28:32,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33806.09 MB 2025-02-15 12:28:32,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:28:32,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30296.74 MB 2025-02-15 12:28:32,606 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 12:28:32,607 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:28:32,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:28:32,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:28:32,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:28:32,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:28:32,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21785.49 MB 2025-02-15 12:28:32,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30177.91 MB 2025-02-15 12:28:32,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-15 12:28:32,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33806.09 MB 2025-02-15 12:28:32,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42148.56 MB 2025-02-15 12:28:32,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 12:28:32,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30177.91 MB 2025-02-15 12:28:32,775 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 12:28:32,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:32,776 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:28:32,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:32,777 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:28:32,782 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:28:32,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:28:32,783 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:28:32,783 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:29:10,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:10,723 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:29:10,730 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:29:10,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:10,737 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1230, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:29:10,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:10,738 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1230, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:29:29,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:29:29,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:29:29,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.07 seconds 2025-02-15 12:29:29,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:29,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21539.54 MB 2025-02-15 12:29:29,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25893.23 MB 2025-02-15 12:29:29,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4353.69 MB 2025-02-15 12:29:29,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50491.03 MB 2025-02-15 12:29:29,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33369.88 MB 2025-02-15 12:29:29,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17121.15 MB 2025-02-15 12:29:29,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34861.28 MB 2025-02-15 12:29:29,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:29:29,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:29:29,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:29:29,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:29,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25893.23 MB 2025-02-15 12:29:29,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22172.24 MB 2025-02-15 12:29:29,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3720.99 MB 2025-02-15 12:29:29,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33369.88 MB 2025-02-15 12:29:29,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42010.15 MB 2025-02-15 12:29:29,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8640.27 MB 2025-02-15 12:29:29,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38887.68 MB 2025-02-15 12:29:31,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:29:31,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:29:31,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:29:31,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:31,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22172.24 MB 2025-02-15 12:29:31,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22703.08 MB 2025-02-15 12:29:31,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:29:31,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42010.15 MB 2025-02-15 12:29:31,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29016.20 MB 2025-02-15 12:29:31,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12993.95 MB 2025-02-15 12:29:31,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26681.63 MB 2025-02-15 12:29:31,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:29:31,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:29:31,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:29:31,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:31,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22703.08 MB 2025-02-15 12:29:31,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24592.61 MB 2025-02-15 12:29:31,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:29:31,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29016.20 MB 2025-02-15 12:29:31,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29016.20 MB 2025-02-15 12:29:31,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:31,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26010.04 MB 2025-02-15 12:29:32,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:29:32,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:29:32,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:29:32,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24592.61 MB 2025-02-15 12:29:32,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26834.47 MB 2025-02-15 12:29:32,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:29:32,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29016.20 MB 2025-02-15 12:29:32,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-15 12:29:32,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:29:32,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32378.75 MB 2025-02-15 12:29:32,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:29:32,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:29:32,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:29:32,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22703.08 MB 2025-02-15 12:29:32,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26834.47 MB 2025-02-15 12:29:32,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:29:32,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29016.20 MB 2025-02-15 12:29:32,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-15 12:29:32,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:29:32,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32378.75 MB 2025-02-15 12:29:32,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:29:32,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:29:32,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:29:32,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28368.01 MB 2025-02-15 12:29:32,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29135.01 MB 2025-02-15 12:29:32,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:29:32,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-15 12:29:32,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-15 12:29:32,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:29:32,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29842.80 MB 2025-02-15 12:29:32,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:29:32,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:29:32,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:29:32,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29547.90 MB 2025-02-15 12:29:32,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29776.74 MB 2025-02-15 12:29:32,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-15 12:29:32,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-15 12:29:32,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-15 12:29:32,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:32,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30021.07 MB 2025-02-15 12:29:32,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:29:32,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:29:32,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.50 seconds 2025-02-15 12:29:32,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17254.12 MB 2025-02-15 12:29:32,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29977.59 MB 2025-02-15 12:29:32,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12723.47 MB 2025-02-15 12:29:32,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50491.03 MB 2025-02-15 12:29:32,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-15 12:29:32,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15397.29 MB 2025-02-15 12:29:32,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30021.07 MB 2025-02-15 12:29:32,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:29:32,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:29:32,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:29:32,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29977.59 MB 2025-02-15 12:29:32,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22254.03 MB 2025-02-15 12:29:32,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7723.56 MB 2025-02-15 12:29:32,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-15 12:29:32,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-15 12:29:32,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:32,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32485.63 MB 2025-02-15 12:29:32,528 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 12:29:32,528 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:29:32,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:29:32,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:29:32,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:29:32,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:32,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22254.03 MB 2025-02-15 12:29:32,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30680.21 MB 2025-02-15 12:29:32,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-15 12:29:32,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-15 12:29:32,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43469.77 MB 2025-02-15 12:29:32,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 12:29:32,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30680.21 MB 2025-02-15 12:29:32,698 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 12:29:32,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:32,699 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:29:32,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:32,700 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:29:32,705 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:29:32,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:32,706 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:29:32,706 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:29:41,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:41,823 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:29:41,827 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:29:41,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:41,831 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 898, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:29:41,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:41,832 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 898, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:29:55,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:29:55,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:29:55,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.04 seconds 2025-02-15 12:29:55,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:55,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19226.11 MB 2025-02-15 12:29:55,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22404.09 MB 2025-02-15 12:29:55,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3177.97 MB 2025-02-15 12:29:55,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51845.79 MB 2025-02-15 12:29:55,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28003.27 MB 2025-02-15 12:29:55,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23842.52 MB 2025-02-15 12:29:55,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31415.39 MB 2025-02-15 12:29:55,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:29:55,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:29:55,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:29:55,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:55,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22404.09 MB 2025-02-15 12:29:55,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20447.32 MB 2025-02-15 12:29:55,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1956.76 MB 2025-02-15 12:29:55,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28003.27 MB 2025-02-15 12:29:55,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35838.23 MB 2025-02-15 12:29:55,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7834.96 MB 2025-02-15 12:29:55,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32340.46 MB 2025-02-15 12:29:57,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:29:57,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:29:57,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:29:57,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:57,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20447.32 MB 2025-02-15 12:29:57,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20978.16 MB 2025-02-15 12:29:57,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:29:57,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35838.23 MB 2025-02-15 12:29:57,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26948.40 MB 2025-02-15 12:29:57,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8889.83 MB 2025-02-15 12:29:57,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24956.71 MB 2025-02-15 12:29:57,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:29:57,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:29:57,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:29:57,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:57,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20978.16 MB 2025-02-15 12:29:57,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22867.70 MB 2025-02-15 12:29:57,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:29:57,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26948.40 MB 2025-02-15 12:29:57,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26948.40 MB 2025-02-15 12:29:57,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:57,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24285.13 MB 2025-02-15 12:29:58,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:29:58,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:29:58,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:29:58,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22867.70 MB 2025-02-15 12:29:58,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25109.55 MB 2025-02-15 12:29:58,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:29:58,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26948.40 MB 2025-02-15 12:29:58,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32610.71 MB 2025-02-15 12:29:58,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:29:58,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.83 MB 2025-02-15 12:29:58,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:29:58,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:29:58,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:29:58,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20978.16 MB 2025-02-15 12:29:58,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25109.55 MB 2025-02-15 12:29:58,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:29:58,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26948.40 MB 2025-02-15 12:29:58,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32610.71 MB 2025-02-15 12:29:58,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:29:58,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.83 MB 2025-02-15 12:29:58,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:29:58,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:29:58,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:29:58,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26643.10 MB 2025-02-15 12:29:58,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27410.10 MB 2025-02-15 12:29:58,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:29:58,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32610.71 MB 2025-02-15 12:29:58,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33023.85 MB 2025-02-15 12:29:58,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 12:29:58,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28117.89 MB 2025-02-15 12:29:58,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:29:58,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:29:58,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:29:58,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27822.99 MB 2025-02-15 12:29:58,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28051.15 MB 2025-02-15 12:29:58,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-15 12:29:58,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33023.85 MB 2025-02-15 12:29:58,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33023.85 MB 2025-02-15 12:29:58,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:58,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28248.92 MB 2025-02-15 12:29:58,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:29:58,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:29:58,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.45 seconds 2025-02-15 12:29:58,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16097.41 MB 2025-02-15 12:29:58,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28252.10 MB 2025-02-15 12:29:58,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12154.69 MB 2025-02-15 12:29:58,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51845.79 MB 2025-02-15 12:29:58,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33023.85 MB 2025-02-15 12:29:58,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18821.94 MB 2025-02-15 12:29:58,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28252.10 MB 2025-02-15 12:29:58,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:29:58,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:29:58,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:29:58,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28252.10 MB 2025-02-15 12:29:58,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21099.89 MB 2025-02-15 12:29:58,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7152.20 MB 2025-02-15 12:29:58,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33023.85 MB 2025-02-15 12:29:58,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33023.85 MB 2025-02-15 12:29:58,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:29:58,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30762.23 MB 2025-02-15 12:29:58,574 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 12:29:58,574 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:29:58,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:29:58,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:29:58,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:29:58,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:29:58,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21099.89 MB 2025-02-15 12:29:58,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29534.51 MB 2025-02-15 12:29:58,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 12:29:58,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33023.85 MB 2025-02-15 12:29:58,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41408.27 MB 2025-02-15 12:29:58,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 12:29:58,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29534.51 MB 2025-02-15 12:29:58,743 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 12:29:58,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:58,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:29:58,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:58,746 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:29:58,751 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:29:58,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:29:58,752 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:29:58,752 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:31:19,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:19,427 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:31:19,432 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:31:19,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:19,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:31:19,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:19,437 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:31:21,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:31:21,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:31:21,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.55 seconds 2025-02-15 12:31:21,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:21,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-15 12:31:21,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-15 12:31:21,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 12:31:21,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49792.68 MB 2025-02-15 12:31:21,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18371.05 MB 2025-02-15 12:31:21,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31421.63 MB 2025-02-15 12:31:21,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-15 12:31:22,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:31:22,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:31:22,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:31:22,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:22,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-15 12:31:22,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14937.06 MB 2025-02-15 12:31:22,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.71 MB 2025-02-15 12:31:22,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18371.05 MB 2025-02-15 12:31:22,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18941.48 MB 2025-02-15 12:31:22,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-15 12:31:22,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16922.48 MB 2025-02-15 12:31:22,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:31:22,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:31:22,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 12:31:22,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:22,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14937.06 MB 2025-02-15 12:31:22,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.72 MB 2025-02-15 12:31:22,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 12:31:22,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18941.48 MB 2025-02-15 12:31:22,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18941.48 MB 2025-02-15 12:31:22,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:31:22,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.81 MB 2025-02-15 12:31:22,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:31:22,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:31:22,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:31:22,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:22,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-15 12:31:22,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15911.01 MB 2025-02-15 12:31:22,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 12:31:22,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18941.48 MB 2025-02-15 12:31:22,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18941.48 MB 2025-02-15 12:31:22,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:31:22,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16481.53 MB 2025-02-15 12:31:22,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:31:22,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:31:22,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:31:22,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:22,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15911.01 MB 2025-02-15 12:31:22,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-15 12:31:22,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 12:31:22,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18941.48 MB 2025-02-15 12:31:22,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20849.89 MB 2025-02-15 12:31:22,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-15 12:31:22,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.93 MB 2025-02-15 12:31:22,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:31:22,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:31:22,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:31:22,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:22,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-15 12:31:22,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-15 12:31:22,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 12:31:22,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18941.48 MB 2025-02-15 12:31:22,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20849.89 MB 2025-02-15 12:31:22,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-15 12:31:22,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.93 MB 2025-02-15 12:31:23,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:31:23,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:31:23,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:31:23,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:23,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17430.65 MB 2025-02-15 12:31:23,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.36 MB 2025-02-15 12:31:23,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 12:31:23,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20849.89 MB 2025-02-15 12:31:23,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21009.27 MB 2025-02-15 12:31:23,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-15 12:31:23,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18031.27 MB 2025-02-15 12:31:23,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:31:23,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:31:23,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:31:23,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:23,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17905.56 MB 2025-02-15 12:31:23,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18133.90 MB 2025-02-15 12:31:23,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.34 MB 2025-02-15 12:31:23,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21009.27 MB 2025-02-15 12:31:23,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21009.27 MB 2025-02-15 12:31:23,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:31:23,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18150.41 MB 2025-02-15 12:31:23,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:31:23,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:31:23,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.64 seconds 2025-02-15 12:31:23,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:23,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-15 12:31:23,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18334.85 MB 2025-02-15 12:31:23,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4805.21 MB 2025-02-15 12:31:23,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49792.68 MB 2025-02-15 12:31:23,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21009.27 MB 2025-02-15 12:31:23,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28783.41 MB 2025-02-15 12:31:23,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18334.85 MB 2025-02-15 12:31:23,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:31:23,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:31:23,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 12:31:23,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:23,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18334.85 MB 2025-02-15 12:31:23,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17404.22 MB 2025-02-15 12:31:23,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -930.63 MB 2025-02-15 12:31:23,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21009.27 MB 2025-02-15 12:31:23,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21009.27 MB 2025-02-15 12:31:23,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:31:23,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19138.09 MB 2025-02-15 12:31:23,401 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 12:31:23,402 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:31:23,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:31:23,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:31:23,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:31:23,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:31:23,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17404.22 MB 2025-02-15 12:31:23,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25838.83 MB 2025-02-15 12:31:23,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 12:31:23,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21009.27 MB 2025-02-15 12:31:23,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31490.83 MB 2025-02-15 12:31:23,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 12:31:23,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25838.83 MB 2025-02-15 12:31:23,669 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 12:31:23,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:23,671 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:31:23,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:23,673 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:31:23,681 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:31:23,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:31:23,683 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:31:23,683 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:32:13,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:13,145 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:32:13,151 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:32:13,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:13,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1737, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:32:13,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:13,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1737, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:32:39,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:32:39,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:32:39,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.77 seconds 2025-02-15 12:32:39,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:39,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25072.40 MB 2025-02-15 12:32:39,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.55 MB 2025-02-15 12:32:39,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6147.15 MB 2025-02-15 12:32:39,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39875.25 MB 2025-02-15 12:32:39,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39397.10 MB 2025-02-15 12:32:39,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -478.15 MB 2025-02-15 12:32:39,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40206.08 MB 2025-02-15 12:32:40,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:32:40,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:32:40,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:32:40,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:40,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31219.55 MB 2025-02-15 12:32:40,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24807.97 MB 2025-02-15 12:32:40,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6411.57 MB 2025-02-15 12:32:40,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39397.10 MB 2025-02-15 12:32:40,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52099.55 MB 2025-02-15 12:32:40,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12702.45 MB 2025-02-15 12:32:40,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48194.87 MB 2025-02-15 12:32:41,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:32:41,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:32:41,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:32:41,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:41,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24807.97 MB 2025-02-15 12:32:41,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25338.81 MB 2025-02-15 12:32:41,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:32:41,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52099.55 MB 2025-02-15 12:32:41,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-15 12:32:41,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17435.72 MB 2025-02-15 12:32:41,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29317.36 MB 2025-02-15 12:32:41,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:32:41,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:32:41,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:32:41,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:41,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-15 12:32:41,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27228.35 MB 2025-02-15 12:32:41,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:32:41,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-15 12:32:41,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-15 12:32:41,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:32:41,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.78 MB 2025-02-15 12:32:42,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:32:42,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:32:42,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:32:42,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27228.35 MB 2025-02-15 12:32:42,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29470.20 MB 2025-02-15 12:32:42,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:32:42,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-15 12:32:42,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37494.98 MB 2025-02-15 12:32:42,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:32:42,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.48 MB 2025-02-15 12:32:42,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:32:42,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:32:42,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:32:42,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-15 12:32:42,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29470.20 MB 2025-02-15 12:32:42,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:32:42,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-15 12:32:42,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37494.98 MB 2025-02-15 12:32:42,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:32:42,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.48 MB 2025-02-15 12:32:42,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:32:42,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:32:42,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:32:42,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.75 MB 2025-02-15 12:32:42,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31770.75 MB 2025-02-15 12:32:42,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:32:42,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37494.98 MB 2025-02-15 12:32:42,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-15 12:32:42,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:32:42,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.54 MB 2025-02-15 12:32:42,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:32:42,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:32:42,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:32:42,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32183.64 MB 2025-02-15 12:32:42,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32411.74 MB 2025-02-15 12:32:42,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-15 12:32:42,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-15 12:32:42,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-15 12:32:42,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:32:42,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32622.92 MB 2025-02-15 12:32:42,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:32:42,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:32:42,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.23 seconds 2025-02-15 12:32:42,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19020.55 MB 2025-02-15 12:32:42,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32612.59 MB 2025-02-15 12:32:42,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13592.04 MB 2025-02-15 12:32:42,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39875.25 MB 2025-02-15 12:32:42,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-15 12:32:42,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1965.03 MB 2025-02-15 12:32:42,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32622.92 MB 2025-02-15 12:32:42,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:32:42,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:32:42,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:32:42,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32612.59 MB 2025-02-15 12:32:42,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24013.32 MB 2025-02-15 12:32:42,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8599.27 MB 2025-02-15 12:32:42,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-15 12:32:42,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-15 12:32:42,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:32:42,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35114.43 MB 2025-02-15 12:32:42,677 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-15 12:32:42,677 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:32:42,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:32:42,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:32:42,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:32:42,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:32:42,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.32 MB 2025-02-15 12:32:42,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32418.98 MB 2025-02-15 12:32:42,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-15 12:32:42,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-15 12:32:42,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42089.84 MB 2025-02-15 12:32:42,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 12:32:42,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32418.98 MB 2025-02-15 12:32:42,841 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-15 12:32:42,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:42,843 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:32:42,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:42,844 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:32:42,848 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:32:42,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:32:42,849 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:32:42,849 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:33:33,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:33,200 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:33:33,205 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:33:33,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:33,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:33:33,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:33,210 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:33:51,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:33:51,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:33:51,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.29 seconds 2025-02-15 12:33:51,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:51,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-15 12:33:51,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-15 12:33:51,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-15 12:33:51,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50449.09 MB 2025-02-15 12:33:51,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 12:33:51,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21422.41 MB 2025-02-15 12:33:51,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34307.29 MB 2025-02-15 12:33:51,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:33:51,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:33:51,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:33:51,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:51,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-15 12:33:51,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21928.95 MB 2025-02-15 12:33:51,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3469.66 MB 2025-02-15 12:33:51,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 12:33:51,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39464.21 MB 2025-02-15 12:33:51,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10437.53 MB 2025-02-15 12:33:51,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37932.68 MB 2025-02-15 12:33:53,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:33:53,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:33:53,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:33:53,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.95 MB 2025-02-15 12:33:53,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22459.79 MB 2025-02-15 12:33:53,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:33:53,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39464.21 MB 2025-02-15 12:33:53,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26963.08 MB 2025-02-15 12:33:53,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12501.12 MB 2025-02-15 12:33:53,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26440.41 MB 2025-02-15 12:33:53,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:33:53,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:33:53,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:33:53,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-15 12:33:53,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24349.32 MB 2025-02-15 12:33:53,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:33:53,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26963.08 MB 2025-02-15 12:33:53,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-15 12:33:53,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:33:53,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25766.75 MB 2025-02-15 12:33:53,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:33:53,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:33:53,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:33:53,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24349.32 MB 2025-02-15 12:33:53,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26592.23 MB 2025-02-15 12:33:53,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-15 12:33:53,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-15 12:33:53,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 12:33:53,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-15 12:33:53,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.51 MB 2025-02-15 12:33:53,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:33:53,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:33:53,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:33:53,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-15 12:33:53,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26592.23 MB 2025-02-15 12:33:53,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-15 12:33:53,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26963.08 MB 2025-02-15 12:33:53,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 12:33:53,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7314.87 MB 2025-02-15 12:33:53,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.51 MB 2025-02-15 12:33:53,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:33:53,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:33:53,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:33:53,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28125.77 MB 2025-02-15 12:33:53,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.77 MB 2025-02-15 12:33:53,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:33:53,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 12:33:53,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 12:33:53,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:33:53,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29600.56 MB 2025-02-15 12:33:53,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:33:53,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:33:53,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:33:53,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29305.66 MB 2025-02-15 12:33:53,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29533.96 MB 2025-02-15 12:33:53,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-15 12:33:53,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34693.19 MB 2025-02-15 12:33:53,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 12:33:53,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:33:53,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29764.25 MB 2025-02-15 12:33:53,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:33:53,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:33:53,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.75 seconds 2025-02-15 12:33:53,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:53,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-15 12:33:53,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29734.81 MB 2025-02-15 12:33:53,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12644.44 MB 2025-02-15 12:33:53,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50449.09 MB 2025-02-15 12:33:53,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 12:33:53,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15755.90 MB 2025-02-15 12:33:53,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29764.25 MB 2025-02-15 12:33:54,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:33:54,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:33:54,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:33:54,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:54,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29734.81 MB 2025-02-15 12:33:54,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22082.07 MB 2025-02-15 12:33:54,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7652.74 MB 2025-02-15 12:33:54,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34693.19 MB 2025-02-15 12:33:54,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34693.19 MB 2025-02-15 12:33:54,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:33:54,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32235.72 MB 2025-02-15 12:33:54,249 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-15 12:33:54,249 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:33:54,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:33:54,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:33:54,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:33:54,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:33:54,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22082.07 MB 2025-02-15 12:33:54,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30485.63 MB 2025-02-15 12:33:54,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-15 12:33:54,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34693.19 MB 2025-02-15 12:33:54,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43048.24 MB 2025-02-15 12:33:54,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 12:33:54,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30485.63 MB 2025-02-15 12:33:54,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-15 12:33:54,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:54,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:33:54,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:54,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:33:54,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:33:54,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:33:54,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:33:54,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:35:25,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:25,261 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:35:25,266 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:35:25,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:25,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:35:25,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:25,271 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:35:42,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:35:42,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:35:42,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.64 seconds 2025-02-15 12:35:42,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:42,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20961.19 MB 2025-02-15 12:35:42,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.27 MB 2025-02-15 12:35:42,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4060.09 MB 2025-02-15 12:35:42,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51403.29 MB 2025-02-15 12:35:42,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28894.56 MB 2025-02-15 12:35:42,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22508.73 MB 2025-02-15 12:35:42,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33829.94 MB 2025-02-15 12:35:43,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:35:43,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:35:43,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:35:43,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:43,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25021.27 MB 2025-02-15 12:35:43,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21741.79 MB 2025-02-15 12:35:43,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3279.48 MB 2025-02-15 12:35:43,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28894.56 MB 2025-02-15 12:35:43,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38711.33 MB 2025-02-15 12:35:43,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9816.77 MB 2025-02-15 12:35:43,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36937.91 MB 2025-02-15 12:35:44,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:35:44,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:35:44,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:35:44,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:44,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21741.79 MB 2025-02-15 12:35:44,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22272.64 MB 2025-02-15 12:35:44,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:35:44,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38711.33 MB 2025-02-15 12:35:44,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26958.89 MB 2025-02-15 12:35:44,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11752.44 MB 2025-02-15 12:35:44,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26251.18 MB 2025-02-15 12:35:44,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:35:44,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:35:44,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:35:44,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:44,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22272.64 MB 2025-02-15 12:35:44,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24162.17 MB 2025-02-15 12:35:44,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:35:44,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26958.89 MB 2025-02-15 12:35:44,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27902.61 MB 2025-02-15 12:35:44,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:35:44,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25579.60 MB 2025-02-15 12:35:45,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:35:45,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:35:45,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:35:45,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24162.17 MB 2025-02-15 12:35:45,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26404.03 MB 2025-02-15 12:35:45,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:35:45,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27902.61 MB 2025-02-15 12:35:45,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34036.78 MB 2025-02-15 12:35:45,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:35:45,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31948.31 MB 2025-02-15 12:35:45,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:35:45,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:35:45,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:35:45,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22272.64 MB 2025-02-15 12:35:45,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26404.03 MB 2025-02-15 12:35:45,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:35:45,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26958.89 MB 2025-02-15 12:35:45,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34036.78 MB 2025-02-15 12:35:45,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 12:35:45,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31948.31 MB 2025-02-15 12:35:45,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:35:45,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:35:45,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:35:45,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27937.57 MB 2025-02-15 12:35:45,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28704.57 MB 2025-02-15 12:35:45,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:35:45,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34036.78 MB 2025-02-15 12:35:45,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-15 12:35:45,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:35:45,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29412.36 MB 2025-02-15 12:35:45,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:35:45,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:35:45,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:35:45,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29117.46 MB 2025-02-15 12:35:45,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29345.85 MB 2025-02-15 12:35:45,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-15 12:35:45,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34452.01 MB 2025-02-15 12:35:45,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-15 12:35:45,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:35:45,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29556.14 MB 2025-02-15 12:35:45,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:35:45,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:35:45,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.09 seconds 2025-02-15 12:35:45,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16964.95 MB 2025-02-15 12:35:45,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29546.17 MB 2025-02-15 12:35:45,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12581.22 MB 2025-02-15 12:35:45,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51403.29 MB 2025-02-15 12:35:45,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-15 12:35:45,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16951.28 MB 2025-02-15 12:35:45,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29556.14 MB 2025-02-15 12:35:45,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:35:45,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:35:45,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:35:45,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29546.17 MB 2025-02-15 12:35:45,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21958.07 MB 2025-02-15 12:35:45,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7588.10 MB 2025-02-15 12:35:45,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34452.01 MB 2025-02-15 12:35:45,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-15 12:35:45,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:35:45,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32048.31 MB 2025-02-15 12:35:45,650 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 12:35:45,651 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:35:45,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:35:45,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:35:45,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:35:45,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:35:45,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21958.07 MB 2025-02-15 12:35:45,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30365.80 MB 2025-02-15 12:35:45,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 12:35:45,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34452.01 MB 2025-02-15 12:35:45,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42811.26 MB 2025-02-15 12:35:45,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 12:35:45,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30365.80 MB 2025-02-15 12:35:45,817 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 12:35:45,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:45,819 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:35:45,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:45,820 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:35:45,824 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:35:45,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:45,825 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:35:45,826 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:35:54,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:54,687 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:35:54,692 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:35:54,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:54,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:35:54,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:35:54,696 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:36:27,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:36:27,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:36:27,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.41 seconds 2025-02-15 12:36:27,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:27,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-15 12:36:27,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-15 12:36:27,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-15 12:36:27,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51170.51 MB 2025-02-15 12:36:27,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40554.73 MB 2025-02-15 12:36:27,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10615.78 MB 2025-02-15 12:36:27,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43721.65 MB 2025-02-15 12:36:27,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:36:27,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:36:27,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:36:27,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:27,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-15 12:36:27,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26586.97 MB 2025-02-15 12:36:27,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8226.00 MB 2025-02-15 12:36:27,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40554.73 MB 2025-02-15 12:36:27,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50633.64 MB 2025-02-15 12:36:27,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10078.91 MB 2025-02-15 12:36:27,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46989.90 MB 2025-02-15 12:36:29,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:36:29,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:36:29,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 12:36:29,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26586.97 MB 2025-02-15 12:36:29,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27117.81 MB 2025-02-15 12:36:29,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:36:29,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50633.64 MB 2025-02-15 12:36:29,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31142.71 MB 2025-02-15 12:36:29,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19490.93 MB 2025-02-15 12:36:29,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31096.36 MB 2025-02-15 12:36:29,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:36:29,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:36:29,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:36:29,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-15 12:36:29,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29007.35 MB 2025-02-15 12:36:29,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:36:29,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31142.71 MB 2025-02-15 12:36:29,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32086.43 MB 2025-02-15 12:36:29,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:36:29,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30424.78 MB 2025-02-15 12:36:29,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:36:29,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:36:29,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:36:29,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.35 MB 2025-02-15 12:36:29,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-15 12:36:29,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:36:29,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-15 12:36:29,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38692.45 MB 2025-02-15 12:36:29,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:36:29,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-15 12:36:29,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:36:29,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:36:29,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.31 seconds 2025-02-15 12:36:29,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-15 12:36:29,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-15 12:36:29,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:36:29,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31142.71 MB 2025-02-15 12:36:29,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38692.45 MB 2025-02-15 12:36:29,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:36:29,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-15 12:36:29,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:36:29,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:36:29,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:36:29,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32782.75 MB 2025-02-15 12:36:29,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33549.75 MB 2025-02-15 12:36:29,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:36:29,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38692.45 MB 2025-02-15 12:36:29,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39105.59 MB 2025-02-15 12:36:29,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 12:36:29,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34257.54 MB 2025-02-15 12:36:29,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:36:29,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:36:29,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:36:29,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33962.64 MB 2025-02-15 12:36:29,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34188.53 MB 2025-02-15 12:36:29,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.89 MB 2025-02-15 12:36:29,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39105.59 MB 2025-02-15 12:36:29,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39105.59 MB 2025-02-15 12:36:29,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:36:29,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34410.74 MB 2025-02-15 12:36:29,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:36:29,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:36:29,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.12 seconds 2025-02-15 12:36:29,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:29,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-15 12:36:29,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34388.76 MB 2025-02-15 12:36:29,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14176.65 MB 2025-02-15 12:36:29,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51170.51 MB 2025-02-15 12:36:29,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39105.59 MB 2025-02-15 12:36:29,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12064.92 MB 2025-02-15 12:36:29,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34410.74 MB 2025-02-15 12:36:30,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:36:30,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:36:30,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 12:36:30,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:30,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34388.76 MB 2025-02-15 12:36:30,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25204.16 MB 2025-02-15 12:36:30,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9184.60 MB 2025-02-15 12:36:30,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39105.59 MB 2025-02-15 12:36:30,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39105.59 MB 2025-02-15 12:36:30,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:36:30,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36890.60 MB 2025-02-15 12:36:30,137 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 12:36:30,137 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:36:30,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:36:30,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:36:30,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:36:30,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:36:30,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25204.16 MB 2025-02-15 12:36:30,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33609.24 MB 2025-02-15 12:36:30,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-15 12:36:30,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39105.59 MB 2025-02-15 12:36:30,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47460.65 MB 2025-02-15 12:36:30,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 12:36:30,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33609.24 MB 2025-02-15 12:36:30,397 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 12:36:30,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:36:30,399 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:36:30,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:36:30,400 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:36:30,406 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:36:30,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:36:30,407 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:36:30,407 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:37:41,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:41,417 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:37:41,422 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:37:41,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:41,426 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 128, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:37:41,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:41,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 128, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:37:43,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:37:43,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:37:43,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 12:37:43,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:43,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13860.63 MB 2025-02-15 12:37:43,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14313.62 MB 2025-02-15 12:37:43,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 452.98 MB 2025-02-15 12:37:43,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55815.70 MB 2025-02-15 12:37:43,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:43,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36752.59 MB 2025-02-15 12:37:43,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23332.00 MB 2025-02-15 12:37:43,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:37:43,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:37:43,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:37:43,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:43,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.62 MB 2025-02-15 12:37:43,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14533.09 MB 2025-02-15 12:37:43,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.47 MB 2025-02-15 12:37:43,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:43,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:43,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:43,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16111.57 MB 2025-02-15 12:37:44,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:37:44,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:37:44,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.61 seconds 2025-02-15 12:37:44,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14533.09 MB 2025-02-15 12:37:44,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.95 MB 2025-02-15 12:37:44,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 169.87 MB 2025-02-15 12:37:44,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:44,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:44,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18703.77 MB 2025-02-15 12:37:44,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:37:44,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:37:44,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:37:44,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.89 MB 2025-02-15 12:37:44,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15307.39 MB 2025-02-15 12:37:44,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 604.50 MB 2025-02-15 12:37:44,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:44,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:44,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15760.98 MB 2025-02-15 12:37:44,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:37:44,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:37:44,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:37:44,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15307.39 MB 2025-02-15 12:37:44,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16024.83 MB 2025-02-15 12:37:44,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 717.44 MB 2025-02-15 12:37:44,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:44,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:44,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17798.96 MB 2025-02-15 12:37:44,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:37:44,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:37:44,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:37:44,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.89 MB 2025-02-15 12:37:44,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16024.83 MB 2025-02-15 12:37:44,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1321.94 MB 2025-02-15 12:37:44,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:44,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19063.11 MB 2025-02-15 12:37:44,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17798.96 MB 2025-02-15 12:37:44,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:37:44,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:37:44,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:37:44,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.56 MB 2025-02-15 12:37:44,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16761.00 MB 2025-02-15 12:37:44,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.44 MB 2025-02-15 12:37:44,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19063.11 MB 2025-02-15 12:37:44,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19191.04 MB 2025-02-15 12:37:44,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 127.93 MB 2025-02-15 12:37:44,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16998.73 MB 2025-02-15 12:37:44,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:37:44,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:37:44,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:37:44,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16893.14 MB 2025-02-15 12:37:44,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17088.50 MB 2025-02-15 12:37:44,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.36 MB 2025-02-15 12:37:44,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19191.04 MB 2025-02-15 12:37:44,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19191.04 MB 2025-02-15 12:37:44,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17088.50 MB 2025-02-15 12:37:44,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:37:44,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:37:44,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.75 seconds 2025-02-15 12:37:44,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13414.67 MB 2025-02-15 12:37:44,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17256.61 MB 2025-02-15 12:37:44,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3841.94 MB 2025-02-15 12:37:44,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55815.70 MB 2025-02-15 12:37:44,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19191.04 MB 2025-02-15 12:37:44,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36624.66 MB 2025-02-15 12:37:44,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17256.61 MB 2025-02-15 12:37:44,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:37:44,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:37:44,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:37:44,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17256.61 MB 2025-02-15 12:37:44,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16625.32 MB 2025-02-15 12:37:44,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -631.29 MB 2025-02-15 12:37:44,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19191.04 MB 2025-02-15 12:37:44,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19191.04 MB 2025-02-15 12:37:44,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:37:44,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18432.62 MB 2025-02-15 12:37:44,414 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6822, cut from 6824 2025-02-15 12:37:44,414 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:37:44,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:37:44,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:37:44,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:37:44,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:37:44,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16625.32 MB 2025-02-15 12:37:44,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23680.88 MB 2025-02-15 12:37:44,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7055.55 MB 2025-02-15 12:37:44,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19191.04 MB 2025-02-15 12:37:44,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27963.42 MB 2025-02-15 12:37:44,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8772.39 MB 2025-02-15 12:37:44,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23680.88 MB 2025-02-15 12:37:44,558 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6614] 2025-02-15 12:37:44,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:44,559 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:37:44,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:44,560 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:37:44,565 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:37:44,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:37:44,566 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:37:44,566 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:38:43,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:38:43,834 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:38:43,840 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:38:43,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:38:43,843 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1572, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:38:43,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:38:43,844 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1572, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:39:08,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:39:08,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:39:08,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.21 seconds 2025-02-15 12:39:08,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:08,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23922.65 MB 2025-02-15 12:39:08,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29486.40 MB 2025-02-15 12:39:08,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5563.74 MB 2025-02-15 12:39:08,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34980.50 MB 2025-02-15 12:39:08,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36073.11 MB 2025-02-15 12:39:08,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1092.62 MB 2025-02-15 12:39:08,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38376.86 MB 2025-02-15 12:39:08,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:39:08,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:39:08,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:39:08,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:08,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29486.40 MB 2025-02-15 12:39:08,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.19 MB 2025-02-15 12:39:08,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5536.21 MB 2025-02-15 12:39:08,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36073.11 MB 2025-02-15 12:39:08,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-15 12:39:08,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11670.65 MB 2025-02-15 12:39:08,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44911.52 MB 2025-02-15 12:39:10,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:39:10,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:39:10,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:39:10,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23950.19 MB 2025-02-15 12:39:10,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24481.03 MB 2025-02-15 12:39:10,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:39:10,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47743.76 MB 2025-02-15 12:39:10,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28416.41 MB 2025-02-15 12:39:10,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19327.35 MB 2025-02-15 12:39:10,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28460.62 MB 2025-02-15 12:39:10,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:39:10,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:39:10,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:39:10,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-15 12:39:10,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.56 MB 2025-02-15 12:39:10,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:39:10,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28416.41 MB 2025-02-15 12:39:10,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30303.85 MB 2025-02-15 12:39:10,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:39:10,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27787.99 MB 2025-02-15 12:39:10,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:39:10,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:39:10,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:39:10,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26370.56 MB 2025-02-15 12:39:10,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-15 12:39:10,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:39:10,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30303.85 MB 2025-02-15 12:39:10,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35968.25 MB 2025-02-15 12:39:10,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 12:39:10,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-15 12:39:10,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:39:10,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:39:10,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:39:10,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-15 12:39:10,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-15 12:39:10,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:39:10,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28416.41 MB 2025-02-15 12:39:10,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35968.25 MB 2025-02-15 12:39:10,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-15 12:39:10,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-15 12:39:10,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:39:10,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:39:10,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:39:10,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30145.96 MB 2025-02-15 12:39:10,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30912.96 MB 2025-02-15 12:39:10,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:39:10,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35968.25 MB 2025-02-15 12:39:10,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36385.59 MB 2025-02-15 12:39:10,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:39:10,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31620.75 MB 2025-02-15 12:39:10,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:39:10,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:39:10,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:39:10,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31325.85 MB 2025-02-15 12:39:10,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31553.84 MB 2025-02-15 12:39:10,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.99 MB 2025-02-15 12:39:10,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36385.59 MB 2025-02-15 12:39:10,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36385.59 MB 2025-02-15 12:39:10,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:39:10,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31758.12 MB 2025-02-15 12:39:10,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:39:10,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:39:10,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.67 seconds 2025-02-15 12:39:10,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18445.68 MB 2025-02-15 12:39:10,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31754.87 MB 2025-02-15 12:39:10,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13309.19 MB 2025-02-15 12:39:10,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34980.50 MB 2025-02-15 12:39:10,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36385.59 MB 2025-02-15 12:39:10,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1405.09 MB 2025-02-15 12:39:10,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31758.12 MB 2025-02-15 12:39:10,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:39:10,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:39:10,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:39:10,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31754.87 MB 2025-02-15 12:39:10,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23449.31 MB 2025-02-15 12:39:10,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8305.56 MB 2025-02-15 12:39:10,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36385.59 MB 2025-02-15 12:39:10,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36385.59 MB 2025-02-15 12:39:10,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:39:10,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.92 MB 2025-02-15 12:39:10,808 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 12:39:10,809 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:39:10,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:39:10,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:39:10,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:39:10,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:39:10,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23449.31 MB 2025-02-15 12:39:10,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31886.78 MB 2025-02-15 12:39:10,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 12:39:10,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36385.59 MB 2025-02-15 12:39:10,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44774.20 MB 2025-02-15 12:39:10,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 12:39:10,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31886.78 MB 2025-02-15 12:39:10,975 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 12:39:10,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:10,976 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:39:10,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:10,977 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:39:10,982 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:39:10,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:10,983 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:39:10,983 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:39:54,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:54,334 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:39:54,338 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:39:54,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:54,342 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1468, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:39:54,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:39:54,343 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1468, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:40:17,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:40:17,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:40:17,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.69 seconds 2025-02-15 12:40:17,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:17,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23197.97 MB 2025-02-15 12:40:17,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28393.14 MB 2025-02-15 12:40:17,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5195.17 MB 2025-02-15 12:40:17,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53162.80 MB 2025-02-15 12:40:17,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-15 12:40:17,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16768.83 MB 2025-02-15 12:40:17,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37199.18 MB 2025-02-15 12:40:17,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:40:17,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:40:17,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:40:17,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:17,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28393.14 MB 2025-02-15 12:40:17,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23409.52 MB 2025-02-15 12:40:17,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4983.61 MB 2025-02-15 12:40:17,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36393.98 MB 2025-02-15 12:40:17,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46200.26 MB 2025-02-15 12:40:17,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9806.28 MB 2025-02-15 12:40:17,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43140.56 MB 2025-02-15 12:40:19,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:40:19,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:40:19,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:40:19,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23409.52 MB 2025-02-15 12:40:19,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23940.37 MB 2025-02-15 12:40:19,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:40:19,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46200.26 MB 2025-02-15 12:40:19,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27688.70 MB 2025-02-15 12:40:19,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18511.56 MB 2025-02-15 12:40:19,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27919.95 MB 2025-02-15 12:40:19,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:40:19,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:40:19,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:40:19,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23940.37 MB 2025-02-15 12:40:19,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25829.90 MB 2025-02-15 12:40:19,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:40:19,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27688.70 MB 2025-02-15 12:40:19,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29576.13 MB 2025-02-15 12:40:19,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:40:19,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27247.33 MB 2025-02-15 12:40:19,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:40:19,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:40:19,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:40:19,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25829.90 MB 2025-02-15 12:40:19,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28071.76 MB 2025-02-15 12:40:19,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:40:19,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29576.13 MB 2025-02-15 12:40:19,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35710.30 MB 2025-02-15 12:40:19,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:40:19,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33616.04 MB 2025-02-15 12:40:19,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:40:19,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:40:19,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:40:19,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23940.37 MB 2025-02-15 12:40:19,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28071.76 MB 2025-02-15 12:40:19,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:40:19,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27688.70 MB 2025-02-15 12:40:19,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35710.30 MB 2025-02-15 12:40:19,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 12:40:19,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33616.04 MB 2025-02-15 12:40:19,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:40:19,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:40:19,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:40:19,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.30 MB 2025-02-15 12:40:19,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30372.30 MB 2025-02-15 12:40:19,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:40:19,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35710.30 MB 2025-02-15 12:40:19,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36127.64 MB 2025-02-15 12:40:19,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 12:40:19,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31080.09 MB 2025-02-15 12:40:19,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:40:19,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:40:19,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:40:19,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30785.19 MB 2025-02-15 12:40:19,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31013.43 MB 2025-02-15 12:40:19,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.24 MB 2025-02-15 12:40:19,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36127.64 MB 2025-02-15 12:40:19,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36127.64 MB 2025-02-15 12:40:19,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:40:19,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31209.68 MB 2025-02-15 12:40:19,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:40:19,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:40:19,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.15 seconds 2025-02-15 12:40:19,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18083.34 MB 2025-02-15 12:40:19,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31213.66 MB 2025-02-15 12:40:19,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13130.33 MB 2025-02-15 12:40:19,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53162.80 MB 2025-02-15 12:40:19,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36127.64 MB 2025-02-15 12:40:19,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17035.17 MB 2025-02-15 12:40:19,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31213.66 MB 2025-02-15 12:40:19,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:40:19,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:40:19,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:40:19,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31213.66 MB 2025-02-15 12:40:19,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23075.39 MB 2025-02-15 12:40:19,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8138.27 MB 2025-02-15 12:40:19,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36127.64 MB 2025-02-15 12:40:19,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36127.64 MB 2025-02-15 12:40:19,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:40:19,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.88 MB 2025-02-15 12:40:19,779 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 12:40:19,780 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:40:19,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:40:19,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:40:19,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:40:19,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:19,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23075.39 MB 2025-02-15 12:40:19,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31480.47 MB 2025-02-15 12:40:19,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-15 12:40:19,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36127.64 MB 2025-02-15 12:40:19,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44482.69 MB 2025-02-15 12:40:19,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 12:40:19,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31480.47 MB 2025-02-15 12:40:19,945 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 12:40:19,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:19,947 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:40:19,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:19,948 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:40:19,952 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:40:19,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:19,953 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:40:19,953 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:40:32,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:32,659 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:40:32,664 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:40:32,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:32,667 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1131, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:40:32,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:32,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1131, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:40:50,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:40:50,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:40:50,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.64 seconds 2025-02-15 12:40:50,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:50,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20849.70 MB 2025-02-15 12:40:50,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24853.16 MB 2025-02-15 12:40:50,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4003.46 MB 2025-02-15 12:40:50,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52837.74 MB 2025-02-15 12:40:50,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31677.48 MB 2025-02-15 12:40:50,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21160.26 MB 2025-02-15 12:40:50,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33718.45 MB 2025-02-15 12:40:50,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:40:50,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:40:50,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:40:50,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:50,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24853.16 MB 2025-02-15 12:40:50,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21657.57 MB 2025-02-15 12:40:50,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3195.59 MB 2025-02-15 12:40:50,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31677.48 MB 2025-02-15 12:40:50,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39518.73 MB 2025-02-15 12:40:50,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7841.25 MB 2025-02-15 12:40:50,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36916.43 MB 2025-02-15 12:40:52,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:40:52,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:40:52,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 12:40:52,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21657.57 MB 2025-02-15 12:40:52,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22188.41 MB 2025-02-15 12:40:52,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:40:52,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39518.73 MB 2025-02-15 12:40:52,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27674.02 MB 2025-02-15 12:40:52,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11844.71 MB 2025-02-15 12:40:52,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.96 MB 2025-02-15 12:40:52,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:40:52,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:40:52,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:40:52,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-15 12:40:52,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24077.94 MB 2025-02-15 12:40:52,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 12:40:52,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27674.02 MB 2025-02-15 12:40:52,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 12:40:52,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:40:52,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25495.37 MB 2025-02-15 12:40:52,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:40:52,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:40:52,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:40:52,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24077.94 MB 2025-02-15 12:40:52,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-15 12:40:52,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:40:52,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28617.74 MB 2025-02-15 12:40:52,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-15 12:40:52,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 12:40:52,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-15 12:40:52,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:40:52,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:40:52,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:40:52,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-15 12:40:52,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-15 12:40:52,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 12:40:52,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27674.02 MB 2025-02-15 12:40:52,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-15 12:40:52,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:40:52,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-15 12:40:52,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:40:52,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:40:52,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:40:52,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27853.34 MB 2025-02-15 12:40:52,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28620.34 MB 2025-02-15 12:40:52,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:40:52,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-15 12:40:52,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-15 12:40:52,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:40:52,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29328.13 MB 2025-02-15 12:40:52,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:40:52,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:40:52,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:40:52,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29033.23 MB 2025-02-15 12:40:52,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29262.09 MB 2025-02-15 12:40:52,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-15 12:40:52,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-15 12:40:52,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-15 12:40:52,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:40:52,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29469.51 MB 2025-02-15 12:40:52,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:40:52,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:40:52,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.07 seconds 2025-02-15 12:40:52,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:52,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16909.20 MB 2025-02-15 12:40:52,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29463.17 MB 2025-02-15 12:40:52,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12553.97 MB 2025-02-15 12:40:52,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52837.74 MB 2025-02-15 12:40:52,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-15 12:40:52,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18614.32 MB 2025-02-15 12:40:52,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29469.51 MB 2025-02-15 12:40:53,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:40:53,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:40:53,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:40:53,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:53,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29463.17 MB 2025-02-15 12:40:53,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21913.59 MB 2025-02-15 12:40:53,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7549.58 MB 2025-02-15 12:40:53,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-15 12:40:53,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-15 12:40:53,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:40:53,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31974.83 MB 2025-02-15 12:40:53,032 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:40:53,032 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:40:53,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:40:53,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:40:53,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:40:53,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:53,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21913.59 MB 2025-02-15 12:40:53,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30352.61 MB 2025-02-15 12:40:53,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:40:53,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-15 12:40:53,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42614.13 MB 2025-02-15 12:40:53,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:40:53,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30352.61 MB 2025-02-15 12:40:53,197 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:40:53,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:53,198 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:40:53,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:53,199 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:40:53,204 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:40:53,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:53,205 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:40:53,205 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:40:54,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:54,540 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:40:54,545 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:40:54,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:54,548 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 274, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:40:54,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:40:54,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 274, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:40:58,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:40:58,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:40:58,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.37 seconds 2025-02-15 12:40:58,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:58,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24401.96 MB 2025-02-15 12:40:58,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25371.63 MB 2025-02-15 12:40:58,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 969.67 MB 2025-02-15 12:40:58,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55199.14 MB 2025-02-15 12:40:58,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31941.72 MB 2025-02-15 12:40:58,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23257.42 MB 2025-02-15 12:40:58,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34326.31 MB 2025-02-15 12:40:58,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:40:58,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:40:58,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:40:58,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:40:58,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15847.59 MB 2025-02-15 12:40:58,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16268.23 MB 2025-02-15 12:40:58,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 420.64 MB 2025-02-15 12:40:58,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31941.72 MB 2025-02-15 12:40:58,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23762.83 MB 2025-02-15 12:40:58,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8178.89 MB 2025-02-15 12:40:58,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19597.95 MB 2025-02-15 12:41:00,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:41:00,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:41:00,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.29 seconds 2025-02-15 12:41:00,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16268.23 MB 2025-02-15 12:41:00,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16622.57 MB 2025-02-15 12:41:00,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.34 MB 2025-02-15 12:41:00,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23762.83 MB 2025-02-15 12:41:00,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22819.11 MB 2025-02-15 12:41:00,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-15 12:41:00,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20607.75 MB 2025-02-15 12:41:00,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:41:00,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:41:00,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:00,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16622.57 MB 2025-02-15 12:41:00,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17883.57 MB 2025-02-15 12:41:00,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.00 MB 2025-02-15 12:41:00,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22819.11 MB 2025-02-15 12:41:00,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22819.11 MB 2025-02-15 12:41:00,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:00,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18829.71 MB 2025-02-15 12:41:00,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:41:00,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:41:00,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:41:00,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.57 MB 2025-02-15 12:41:00,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.03 MB 2025-02-15 12:41:00,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1496.46 MB 2025-02-15 12:41:00,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22819.11 MB 2025-02-15 12:41:00,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24398.27 MB 2025-02-15 12:41:00,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1579.16 MB 2025-02-15 12:41:00,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23080.82 MB 2025-02-15 12:41:00,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:41:00,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:41:00,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:41:00,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16622.57 MB 2025-02-15 12:41:00,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.03 MB 2025-02-15 12:41:00,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2757.46 MB 2025-02-15 12:41:00,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22819.11 MB 2025-02-15 12:41:00,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24398.27 MB 2025-02-15 12:41:00,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1579.16 MB 2025-02-15 12:41:00,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23080.82 MB 2025-02-15 12:41:00,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:41:00,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:41:00,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:41:00,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20403.67 MB 2025-02-15 12:41:00,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20915.64 MB 2025-02-15 12:41:00,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 511.97 MB 2025-02-15 12:41:00,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24398.27 MB 2025-02-15 12:41:00,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24672.99 MB 2025-02-15 12:41:00,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 274.73 MB 2025-02-15 12:41:00,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21388.09 MB 2025-02-15 12:41:00,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:41:00,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:41:00,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:00,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21191.25 MB 2025-02-15 12:41:00,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21400.56 MB 2025-02-15 12:41:00,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.31 MB 2025-02-15 12:41:00,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24672.99 MB 2025-02-15 12:41:00,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24675.09 MB 2025-02-15 12:41:00,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 12:41:00,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21498.98 MB 2025-02-15 12:41:00,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:41:00,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:41:00,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.98 seconds 2025-02-15 12:41:00,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23447.32 MB 2025-02-15 12:41:00,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21601.63 MB 2025-02-15 12:41:00,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1845.69 MB 2025-02-15 12:41:00,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55199.14 MB 2025-02-15 12:41:00,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24675.09 MB 2025-02-15 12:41:00,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30524.05 MB 2025-02-15 12:41:00,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21601.63 MB 2025-02-15 12:41:00,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:41:00,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:41:00,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:41:00,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21601.63 MB 2025-02-15 12:41:00,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24615.67 MB 2025-02-15 12:41:00,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 12:41:00,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24675.09 MB 2025-02-15 12:41:00,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26017.27 MB 2025-02-15 12:41:00,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-15 12:41:00,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24917.30 MB 2025-02-15 12:41:00,818 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:41:00,818 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:41:00,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:41:00,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:41:00,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:41:00,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:00,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18300.03 MB 2025-02-15 12:41:00,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26739.05 MB 2025-02-15 12:41:00,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 12:41:00,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26017.27 MB 2025-02-15 12:41:00,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36507.22 MB 2025-02-15 12:41:00,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:41:00,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26739.05 MB 2025-02-15 12:41:00,986 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:41:00,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:00,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:00,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:00,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:41:00,993 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:41:00,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:00,994 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:41:00,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:41:02,066 - trainer.py:3503 - _save - INFO - Saving model checkpoint to ./checkpoints/cambrian_llama3_2/checkpoint-540 2025-02-15 12:41:02,068 - configuration_utils.py:472 - save_pretrained - INFO - Configuration saved in ./checkpoints/cambrian_llama3_2/checkpoint-540/config.json 2025-02-15 12:41:02,069 - configuration_utils.py:807 - save_pretrained - INFO - Configuration saved in ./checkpoints/cambrian_llama3_2/checkpoint-540/generation_config.json 2025-02-15 12:41:07,693 - modeling_utils.py:2750 - save_pretrained - INFO - The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 2 checkpoint shards. You can find where each parameters has been saved in the index located at ./checkpoints/cambrian_llama3_2/checkpoint-540/model.safetensors.index.json. 2025-02-15 12:41:07,696 - tokenization_utils_base.py:2702 - save_pretrained - INFO - tokenizer config file saved in ./checkpoints/cambrian_llama3_2/checkpoint-540/tokenizer_config.json 2025-02-15 12:41:07,696 - tokenization_utils_base.py:2711 - save_pretrained - INFO - Special tokens file saved in ./checkpoints/cambrian_llama3_2/checkpoint-540/special_tokens_map.json 2025-02-15 12:41:32,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:32,548 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:32,553 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:41:32,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:32,554 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:41:32,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:32,555 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:41:35,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:41:35,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:41:35,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-15 12:41:35,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:35,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15282.34 MB 2025-02-15 12:41:35,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15972.44 MB 2025-02-15 12:41:35,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-15 12:41:35,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49092.23 MB 2025-02-15 12:41:35,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21265.12 MB 2025-02-15 12:41:35,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27827.11 MB 2025-02-15 12:41:35,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24980.21 MB 2025-02-15 12:41:35,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:41:35,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:41:35,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:35,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:35,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15972.44 MB 2025-02-15 12:41:35,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16292.74 MB 2025-02-15 12:41:35,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.30 MB 2025-02-15 12:41:35,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21265.12 MB 2025-02-15 12:41:35,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21265.12 MB 2025-02-15 12:41:35,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:35,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18683.39 MB 2025-02-15 12:41:36,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:41:36,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:41:36,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 12:41:36,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16292.74 MB 2025-02-15 12:41:36,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16548.87 MB 2025-02-15 12:41:36,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 12:41:36,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21265.12 MB 2025-02-15 12:41:36,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18996.00 MB 2025-02-15 12:41:36,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2269.12 MB 2025-02-15 12:41:36,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20548.36 MB 2025-02-15 12:41:36,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:41:36,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:41:36,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:36,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16548.81 MB 2025-02-15 12:41:36,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17460.28 MB 2025-02-15 12:41:36,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 12:41:36,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18996.00 MB 2025-02-15 12:41:36,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19453.18 MB 2025-02-15 12:41:36,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-15 12:41:36,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18144.20 MB 2025-02-15 12:41:36,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:41:36,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:41:36,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:41:36,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17460.28 MB 2025-02-15 12:41:36,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18542.01 MB 2025-02-15 12:41:36,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 12:41:36,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19453.18 MB 2025-02-15 12:41:36,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22424.85 MB 2025-02-15 12:41:36,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2971.66 MB 2025-02-15 12:41:36,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21218.01 MB 2025-02-15 12:41:36,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:41:36,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:41:36,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:41:36,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16548.81 MB 2025-02-15 12:41:36,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18542.01 MB 2025-02-15 12:41:36,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 12:41:36,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18996.00 MB 2025-02-15 12:41:36,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22424.85 MB 2025-02-15 12:41:36,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3428.84 MB 2025-02-15 12:41:36,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21218.01 MB 2025-02-15 12:41:36,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:41:36,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:41:36,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:41:36,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18883.52 MB 2025-02-15 12:41:36,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19254.52 MB 2025-02-15 12:41:36,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.00 MB 2025-02-15 12:41:36,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22424.85 MB 2025-02-15 12:41:36,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22626.17 MB 2025-02-15 12:41:36,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-15 12:41:36,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19600.53 MB 2025-02-15 12:41:36,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:41:36,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:41:36,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:36,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19453.74 MB 2025-02-15 12:41:36,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19660.27 MB 2025-02-15 12:41:36,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.53 MB 2025-02-15 12:41:36,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22626.17 MB 2025-02-15 12:41:36,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22626.17 MB 2025-02-15 12:41:36,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:36,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19691.93 MB 2025-02-15 12:41:36,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:41:36,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:41:36,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.22 seconds 2025-02-15 12:41:36,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:36,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14602.95 MB 2025-02-15 12:41:36,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19861.34 MB 2025-02-15 12:41:36,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5258.40 MB 2025-02-15 12:41:36,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49092.23 MB 2025-02-15 12:41:36,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22626.17 MB 2025-02-15 12:41:36,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26466.06 MB 2025-02-15 12:41:36,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19861.34 MB 2025-02-15 12:41:37,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:41:37,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:41:37,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:41:37,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:37,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19861.34 MB 2025-02-15 12:41:37,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19961.81 MB 2025-02-15 12:41:37,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:41:37,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22626.17 MB 2025-02-15 12:41:37,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22626.17 MB 2025-02-15 12:41:37,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:37,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20564.61 MB 2025-02-15 12:41:37,055 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:41:37,056 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:41:37,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:41:37,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:41:37,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:41:37,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:37,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15317.25 MB 2025-02-15 12:41:37,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19511.73 MB 2025-02-15 12:41:37,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:41:37,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22626.17 MB 2025-02-15 12:41:37,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33116.13 MB 2025-02-15 12:41:37,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:41:37,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23706.04 MB 2025-02-15 12:41:37,219 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:41:37,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,221 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,221 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:41:37,226 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:41:37,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,227 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:41:37,227 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:41:37,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,228 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,228 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:37,234 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:41:37,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,235 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,235 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:37,235 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:41:37,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,236 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:37,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,236 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:41:37,236 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:41:37,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,237 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:37,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,239 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,240 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,241 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:37,242 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:37,242 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:44,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:44,528 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:44,533 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:41:44,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:44,534 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 261, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:41:44,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:44,535 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 261, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:41:48,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:41:48,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:41:48,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.07 seconds 2025-02-15 12:41:48,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:48,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15863.97 MB 2025-02-15 12:41:48,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16787.63 MB 2025-02-15 12:41:48,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 923.66 MB 2025-02-15 12:41:48,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33116.13 MB 2025-02-15 12:41:48,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-15 12:41:48,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12025.07 MB 2025-02-15 12:41:48,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25788.32 MB 2025-02-15 12:41:48,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:41:48,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:41:48,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:41:48,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:48,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16787.63 MB 2025-02-15 12:41:48,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17087.60 MB 2025-02-15 12:41:48,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.96 MB 2025-02-15 12:41:48,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-15 12:41:48,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21940.40 MB 2025-02-15 12:41:48,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 849.35 MB 2025-02-15 12:41:48,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20158.68 MB 2025-02-15 12:41:49,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:41:49,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:41:49,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.14 seconds 2025-02-15 12:41:49,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:49,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17087.60 MB 2025-02-15 12:41:49,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.10 MB 2025-02-15 12:41:49,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.50 MB 2025-02-15 12:41:49,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21940.40 MB 2025-02-15 12:41:49,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21940.40 MB 2025-02-15 12:41:49,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:49,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21343.22 MB 2025-02-15 12:41:49,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:41:49,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:41:49,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:49,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:49,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.10 MB 2025-02-15 12:41:49,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18539.55 MB 2025-02-15 12:41:49,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1133.45 MB 2025-02-15 12:41:49,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21940.40 MB 2025-02-15 12:41:49,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21940.40 MB 2025-02-15 12:41:49,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:49,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19390.01 MB 2025-02-15 12:41:49,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:41:49,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:41:49,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:41:49,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:49,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18539.55 MB 2025-02-15 12:41:49,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19884.69 MB 2025-02-15 12:41:49,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1345.14 MB 2025-02-15 12:41:49,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21940.40 MB 2025-02-15 12:41:49,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25054.67 MB 2025-02-15 12:41:49,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3114.27 MB 2025-02-15 12:41:49,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23211.23 MB 2025-02-15 12:41:49,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:41:49,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:41:49,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:41:49,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:49,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.10 MB 2025-02-15 12:41:49,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19884.69 MB 2025-02-15 12:41:49,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2478.58 MB 2025-02-15 12:41:49,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21940.40 MB 2025-02-15 12:41:49,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25054.67 MB 2025-02-15 12:41:49,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3114.27 MB 2025-02-15 12:41:49,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23211.23 MB 2025-02-15 12:41:50,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:41:50,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:41:50,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:41:50,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:50,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20309.36 MB 2025-02-15 12:41:50,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20769.56 MB 2025-02-15 12:41:50,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 460.20 MB 2025-02-15 12:41:50,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25054.67 MB 2025-02-15 12:41:50,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25304.24 MB 2025-02-15 12:41:50,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 249.56 MB 2025-02-15 12:41:50,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21194.23 MB 2025-02-15 12:41:50,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:41:50,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:41:50,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:41:50,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:50,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21017.30 MB 2025-02-15 12:41:50,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21223.83 MB 2025-02-15 12:41:50,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.53 MB 2025-02-15 12:41:50,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25304.24 MB 2025-02-15 12:41:50,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25304.24 MB 2025-02-15 12:41:50,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:50,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21289.64 MB 2025-02-15 12:41:50,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:41:50,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:41:50,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.48 seconds 2025-02-15 12:41:50,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:50,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.62 MB 2025-02-15 12:41:50,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21424.83 MB 2025-02-15 12:41:50,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6470.21 MB 2025-02-15 12:41:50,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33116.13 MB 2025-02-15 12:41:50,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25304.24 MB 2025-02-15 12:41:50,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7811.89 MB 2025-02-15 12:41:50,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21424.83 MB 2025-02-15 12:41:50,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:41:50,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:41:50,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:41:50,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:50,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21424.83 MB 2025-02-15 12:41:50,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21525.26 MB 2025-02-15 12:41:50,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.43 MB 2025-02-15 12:41:50,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25304.24 MB 2025-02-15 12:41:50,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25304.24 MB 2025-02-15 12:41:50,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:41:50,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22127.84 MB 2025-02-15 12:41:50,295 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 12:41:50,295 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:41:50,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:41:50,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:41:50,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:41:50,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:41:50,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21525.26 MB 2025-02-15 12:41:50,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19985.65 MB 2025-02-15 12:41:50,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1539.61 MB 2025-02-15 12:41:50,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25304.24 MB 2025-02-15 12:41:50,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35790.00 MB 2025-02-15 12:41:50,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 12:41:50,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24178.08 MB 2025-02-15 12:41:50,462 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 12:41:50,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,463 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,464 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:41:50,468 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:41:50,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,469 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:41:50,470 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:41:50,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,470 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,471 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:50,477 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:41:50,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,477 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,478 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:50,478 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:41:50,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,478 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:50,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,479 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:41:50,479 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:41:50,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,479 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:41:50,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,482 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,483 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,484 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:41:50,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:41:50,485 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:00,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:00,151 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:00,155 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:42:00,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:00,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 148, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:42:00,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:00,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 148, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:42:02,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:42:02,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:42:02,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.31 seconds 2025-02-15 12:42:02,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:02,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15320.02 MB 2025-02-15 12:42:02,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15843.78 MB 2025-02-15 12:42:02,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 523.76 MB 2025-02-15 12:42:02,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35790.00 MB 2025-02-15 12:42:02,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19738.39 MB 2025-02-15 12:42:02,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16051.60 MB 2025-02-15 12:42:02,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24791.39 MB 2025-02-15 12:42:02,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:42:02,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:42:02,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:02,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:02,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15843.78 MB 2025-02-15 12:42:02,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16076.47 MB 2025-02-15 12:42:02,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.69 MB 2025-02-15 12:42:02,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19738.39 MB 2025-02-15 12:42:02,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19738.39 MB 2025-02-15 12:42:02,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:02,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17880.52 MB 2025-02-15 12:42:03,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:42:03,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:42:03,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-15 12:42:03,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16076.47 MB 2025-02-15 12:42:03,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16268.90 MB 2025-02-15 12:42:03,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 12:42:03,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19738.39 MB 2025-02-15 12:42:03,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19455.28 MB 2025-02-15 12:42:03,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -283.12 MB 2025-02-15 12:42:03,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20247.16 MB 2025-02-15 12:42:03,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:42:03,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:42:03,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:42:03,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16268.84 MB 2025-02-15 12:42:03,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16953.63 MB 2025-02-15 12:42:03,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 12:42:03,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19455.28 MB 2025-02-15 12:42:03,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19455.28 MB 2025-02-15 12:42:03,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:03,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17467.45 MB 2025-02-15 12:42:03,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:42:03,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:42:03,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:42:03,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16953.63 MB 2025-02-15 12:42:03,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17766.34 MB 2025-02-15 12:42:03,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 12:42:03,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19455.28 MB 2025-02-15 12:42:03,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20831.01 MB 2025-02-15 12:42:03,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 12:42:03,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19776.10 MB 2025-02-15 12:42:03,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:42:03,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:42:03,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:42:03,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16268.84 MB 2025-02-15 12:42:03,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17766.34 MB 2025-02-15 12:42:03,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 12:42:03,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19455.28 MB 2025-02-15 12:42:03,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20831.01 MB 2025-02-15 12:42:03,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-15 12:42:03,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19776.10 MB 2025-02-15 12:42:03,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:42:03,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:42:03,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:42:03,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18022.92 MB 2025-02-15 12:42:03,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18300.96 MB 2025-02-15 12:42:03,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 12:42:03,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20831.01 MB 2025-02-15 12:42:03,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20979.91 MB 2025-02-15 12:42:03,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-15 12:42:03,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18568.70 MB 2025-02-15 12:42:03,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:42:03,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:42:03,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:03,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18450.64 MB 2025-02-15 12:42:03,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18655.78 MB 2025-02-15 12:42:03,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.14 MB 2025-02-15 12:42:03,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20979.91 MB 2025-02-15 12:42:03,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20979.91 MB 2025-02-15 12:42:03,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:03,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18655.78 MB 2025-02-15 12:42:03,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:42:03,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:42:03,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.18 seconds 2025-02-15 12:42:03,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14804.37 MB 2025-02-15 12:42:03,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18856.78 MB 2025-02-15 12:42:03,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4052.40 MB 2025-02-15 12:42:03,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35790.00 MB 2025-02-15 12:42:03,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20979.91 MB 2025-02-15 12:42:03,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14810.09 MB 2025-02-15 12:42:03,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18856.78 MB 2025-02-15 12:42:03,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:42:03,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:42:03,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:42:03,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18856.78 MB 2025-02-15 12:42:03,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18957.21 MB 2025-02-15 12:42:03,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.43 MB 2025-02-15 12:42:03,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20979.91 MB 2025-02-15 12:42:03,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20979.91 MB 2025-02-15 12:42:03,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:03,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19559.78 MB 2025-02-15 12:42:03,622 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 12:42:03,622 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:03,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:42:03,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:42:03,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:42:03,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:03,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15390.25 MB 2025-02-15 12:42:03,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19583.20 MB 2025-02-15 12:42:03,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.95 MB 2025-02-15 12:42:03,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20979.91 MB 2025-02-15 12:42:03,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31465.67 MB 2025-02-15 12:42:03,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 12:42:03,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23775.63 MB 2025-02-15 12:42:03,791 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 12:42:03,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,792 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,793 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:42:03,798 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:42:03,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,799 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:42:03,799 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:03,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,800 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,800 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:03,806 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:42:03,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,806 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,807 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:03,807 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:42:03,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,807 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:03,808 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,808 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:42:03,808 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:42:03,808 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,808 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:03,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,811 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,812 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,813 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:03,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:03,814 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:13,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:13,950 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:13,955 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:42:13,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:13,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:42:13,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:13,957 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:42:17,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:42:17,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:42:17,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.05 seconds 2025-02-15 12:42:17,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15769.25 MB 2025-02-15 12:42:17,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16459.34 MB 2025-02-15 12:42:17,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-15 12:42:17,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31465.67 MB 2025-02-15 12:42:17,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 12:42:17,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10317.99 MB 2025-02-15 12:42:17,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25467.11 MB 2025-02-15 12:42:17,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:42:17,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:42:17,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:17,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16459.34 MB 2025-02-15 12:42:17,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16674.23 MB 2025-02-15 12:42:17,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.89 MB 2025-02-15 12:42:17,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 12:42:17,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-15 12:42:17,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:17,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18959.54 MB 2025-02-15 12:42:17,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:42:17,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:42:17,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 12:42:17,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16674.23 MB 2025-02-15 12:42:17,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16910.46 MB 2025-02-15 12:42:17,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-15 12:42:17,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-15 12:42:17,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20690.50 MB 2025-02-15 12:42:17,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -457.18 MB 2025-02-15 12:42:17,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20844.92 MB 2025-02-15 12:42:17,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:42:17,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:42:17,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:17,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16910.46 MB 2025-02-15 12:42:17,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17751.10 MB 2025-02-15 12:42:17,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-15 12:42:17,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20690.50 MB 2025-02-15 12:42:17,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20690.50 MB 2025-02-15 12:42:17,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:17,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18381.86 MB 2025-02-15 12:42:17,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:42:17,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:42:17,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:42:17,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17751.10 MB 2025-02-15 12:42:17,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18748.76 MB 2025-02-15 12:42:17,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-15 12:42:17,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20690.50 MB 2025-02-15 12:42:17,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22798.14 MB 2025-02-15 12:42:17,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2107.64 MB 2025-02-15 12:42:17,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21215.93 MB 2025-02-15 12:42:17,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:42:17,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:42:17,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:42:17,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:17,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16910.46 MB 2025-02-15 12:42:17,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18748.76 MB 2025-02-15 12:42:17,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-15 12:42:17,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20690.50 MB 2025-02-15 12:42:17,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22798.14 MB 2025-02-15 12:42:17,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2107.64 MB 2025-02-15 12:42:17,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21215.93 MB 2025-02-15 12:42:18,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:42:18,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:42:18,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:42:18,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:18,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19063.73 MB 2025-02-15 12:42:18,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19405.04 MB 2025-02-15 12:42:18,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.32 MB 2025-02-15 12:42:18,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22798.14 MB 2025-02-15 12:42:18,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22982.69 MB 2025-02-15 12:42:18,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-15 12:42:18,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19726.77 MB 2025-02-15 12:42:18,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:42:18,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:42:18,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:18,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:18,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19588.78 MB 2025-02-15 12:42:18,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19790.26 MB 2025-02-15 12:42:18,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.48 MB 2025-02-15 12:42:18,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22982.69 MB 2025-02-15 12:42:18,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-15 12:42:18,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:42:18,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19820.16 MB 2025-02-15 12:42:18,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:42:18,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:42:18,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.10 seconds 2025-02-15 12:42:18,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:18,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15089.85 MB 2025-02-15 12:42:18,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19991.33 MB 2025-02-15 12:42:18,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.48 MB 2025-02-15 12:42:18,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31465.67 MB 2025-02-15 12:42:18,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-15 12:42:18,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8478.79 MB 2025-02-15 12:42:18,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19991.33 MB 2025-02-15 12:42:18,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:42:18,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:42:18,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:42:18,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:18,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19991.33 MB 2025-02-15 12:42:18,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20091.80 MB 2025-02-15 12:42:18,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:42:18,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-15 12:42:18,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-15 12:42:18,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:18,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20694.60 MB 2025-02-15 12:42:18,347 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:42:18,347 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:18,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:42:18,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:42:18,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:42:18,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:18,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15763.41 MB 2025-02-15 12:42:18,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19957.90 MB 2025-02-15 12:42:18,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:42:18,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-15 12:42:18,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33476.84 MB 2025-02-15 12:42:18,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:42:18,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24152.20 MB 2025-02-15 12:42:18,510 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:42:18,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,511 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,512 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:42:18,516 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:42:18,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,517 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:42:18,518 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:18,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,518 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,519 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:18,524 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:42:18,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,525 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,526 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:18,526 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:42:18,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,526 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:18,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,526 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:42:18,527 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:42:18,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,527 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:18,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,530 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,531 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,532 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:18,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:18,533 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:25,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:25,011 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:25,016 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:42:25,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:25,017 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 206, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:42:25,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:25,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 206, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:42:28,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:42:28,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:42:28,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-15 12:42:28,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:28,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15967.87 MB 2025-02-15 12:42:28,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16696.89 MB 2025-02-15 12:42:28,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 729.02 MB 2025-02-15 12:42:28,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33476.84 MB 2025-02-15 12:42:28,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 12:42:28,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12631.15 MB 2025-02-15 12:42:28,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25665.74 MB 2025-02-15 12:42:28,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:42:28,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:42:28,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:28,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:28,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16696.89 MB 2025-02-15 12:42:28,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16958.74 MB 2025-02-15 12:42:28,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 261.84 MB 2025-02-15 12:42:28,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20845.69 MB 2025-02-15 12:42:28,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20845.69 MB 2025-02-15 12:42:28,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:28,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19407.78 MB 2025-02-15 12:42:29,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:42:29,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:42:29,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-15 12:42:29,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16958.74 MB 2025-02-15 12:42:29,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17214.87 MB 2025-02-15 12:42:29,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 12:42:29,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20845.69 MB 2025-02-15 12:42:29,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20826.82 MB 2025-02-15 12:42:29,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18.87 MB 2025-02-15 12:42:29,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21214.36 MB 2025-02-15 12:42:29,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:42:29,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:42:29,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:29,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17214.87 MB 2025-02-15 12:42:29,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18126.35 MB 2025-02-15 12:42:29,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 12:42:29,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20826.82 MB 2025-02-15 12:42:29,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20826.82 MB 2025-02-15 12:42:29,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:29,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18810.26 MB 2025-02-15 12:42:29,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:42:29,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:42:29,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:42:29,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18126.35 MB 2025-02-15 12:42:29,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19208.08 MB 2025-02-15 12:42:29,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 12:42:29,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20826.82 MB 2025-02-15 12:42:29,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23112.71 MB 2025-02-15 12:42:29,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 12:42:29,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21883.16 MB 2025-02-15 12:42:29,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:42:29,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:42:29,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:42:29,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17214.87 MB 2025-02-15 12:42:29,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19208.08 MB 2025-02-15 12:42:29,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 12:42:29,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20826.82 MB 2025-02-15 12:42:29,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23112.71 MB 2025-02-15 12:42:29,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 12:42:29,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21883.16 MB 2025-02-15 12:42:29,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:42:29,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:42:29,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:42:29,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19549.59 MB 2025-02-15 12:42:29,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19919.67 MB 2025-02-15 12:42:29,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 12:42:29,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23112.71 MB 2025-02-15 12:42:29,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 12:42:29,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-15 12:42:29,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20265.39 MB 2025-02-15 12:42:29,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:42:29,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:42:29,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:29,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20118.89 MB 2025-02-15 12:42:29,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20325.53 MB 2025-02-15 12:42:29,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.64 MB 2025-02-15 12:42:29,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23314.04 MB 2025-02-15 12:42:29,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 12:42:29,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:29,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20361.69 MB 2025-02-15 12:42:29,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:42:29,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:42:29,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.37 seconds 2025-02-15 12:42:29,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15250.15 MB 2025-02-15 12:42:29,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20526.60 MB 2025-02-15 12:42:29,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5276.45 MB 2025-02-15 12:42:29,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33476.84 MB 2025-02-15 12:42:29,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 12:42:29,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10162.80 MB 2025-02-15 12:42:29,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20526.60 MB 2025-02-15 12:42:29,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:42:29,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:42:29,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:42:29,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20526.60 MB 2025-02-15 12:42:29,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20627.07 MB 2025-02-15 12:42:29,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:42:29,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23314.04 MB 2025-02-15 12:42:29,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23314.04 MB 2025-02-15 12:42:29,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:29,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21229.87 MB 2025-02-15 12:42:29,695 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:42:29,696 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:29,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:42:29,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:42:29,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:42:29,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:29,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15963.53 MB 2025-02-15 12:42:29,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20158.02 MB 2025-02-15 12:42:29,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:42:29,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23314.04 MB 2025-02-15 12:42:29,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 12:42:29,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:42:29,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24352.32 MB 2025-02-15 12:42:29,861 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:42:29,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,862 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,863 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:42:29,868 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:42:29,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,869 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:42:29,869 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:42:29,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,870 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,870 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:29,876 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:42:29,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,877 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,877 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:29,877 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:42:29,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,878 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:29,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,878 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:42:29,878 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:42:29,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,879 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:29,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,882 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,882 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,883 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:29,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:29,884 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:40,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:40,399 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:40,404 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:42:40,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:40,405 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 100, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:42:40,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:40,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 100, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:42:41,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:42:41,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:42:41,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-15 12:42:41,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:41,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15350.72 MB 2025-02-15 12:42:41,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15704.62 MB 2025-02-15 12:42:41,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.89 MB 2025-02-15 12:42:41,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 12:42:41,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:41,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14579.40 MB 2025-02-15 12:42:41,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24595.60 MB 2025-02-15 12:42:41,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:42:41,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:42:41,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:42:41,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:41,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15704.62 MB 2025-02-15 12:42:41,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15876.08 MB 2025-02-15 12:42:41,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 171.46 MB 2025-02-15 12:42:41,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:41,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:41,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:41,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16406.99 MB 2025-02-15 12:42:42,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:42:42,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:42:42,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.48 seconds 2025-02-15 12:42:42,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15876.08 MB 2025-02-15 12:42:42,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16008.79 MB 2025-02-15 12:42:42,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 132.71 MB 2025-02-15 12:42:42,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:42,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:42,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19961.83 MB 2025-02-15 12:42:42,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:42:42,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:42:42,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:42:42,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16008.72 MB 2025-02-15 12:42:42,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16480.99 MB 2025-02-15 12:42:42,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.27 MB 2025-02-15 12:42:42,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:42,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:42,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16835.36 MB 2025-02-15 12:42:42,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:42:42,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:42:42,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:42:42,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16480.99 MB 2025-02-15 12:42:42,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17054.61 MB 2025-02-15 12:42:42,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 573.62 MB 2025-02-15 12:42:42,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:42,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:42,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18427.53 MB 2025-02-15 12:42:42,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:42:42,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:42:42,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:42:42,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16008.72 MB 2025-02-15 12:42:42,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17054.61 MB 2025-02-15 12:42:42,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1045.89 MB 2025-02-15 12:42:42,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:42,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-15 12:42:42,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18427.53 MB 2025-02-15 12:42:42,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:42:42,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:42:42,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 12:42:42,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17310.20 MB 2025-02-15 12:42:42,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17551.11 MB 2025-02-15 12:42:42,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.90 MB 2025-02-15 12:42:42,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-15 12:42:42,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19379.78 MB 2025-02-15 12:42:42,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 155.19 MB 2025-02-15 12:42:42,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17728.05 MB 2025-02-15 12:42:42,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:42:42,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:42:42,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:42:42,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17703.49 MB 2025-02-15 12:42:42,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17903.09 MB 2025-02-15 12:42:42,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.60 MB 2025-02-15 12:42:42,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19379.78 MB 2025-02-15 12:42:42,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19379.78 MB 2025-02-15 12:42:42,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17903.09 MB 2025-02-15 12:42:42,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:42:42,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:42:42,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.22 seconds 2025-02-15 12:42:42,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15002.32 MB 2025-02-15 12:42:42,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18103.60 MB 2025-02-15 12:42:42,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3101.28 MB 2025-02-15 12:42:42,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 12:42:42,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19379.78 MB 2025-02-15 12:42:42,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14424.21 MB 2025-02-15 12:42:42,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18103.60 MB 2025-02-15 12:42:42,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:42:42,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:42:42,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:42:42,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15368.05 MB 2025-02-15 12:42:42,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15468.24 MB 2025-02-15 12:42:42,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.18 MB 2025-02-15 12:42:42,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19379.78 MB 2025-02-15 12:42:42,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19379.78 MB 2025-02-15 12:42:42,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:42,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16069.91 MB 2025-02-15 12:42:42,912 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 12:42:42,912 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:42:42,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:42:42,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:42:42,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:42:42,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:42,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15468.24 MB 2025-02-15 12:42:42,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19650.92 MB 2025-02-15 12:42:42,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4182.69 MB 2025-02-15 12:42:42,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19379.78 MB 2025-02-15 12:42:42,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29840.38 MB 2025-02-15 12:42:42,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-15 12:42:42,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23833.10 MB 2025-02-15 12:42:43,078 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 12:42:43,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,080 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,081 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:42:43,085 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:42:43,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,086 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:42:43,086 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:42:43,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,087 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,088 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:43,094 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:42:43,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,094 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,095 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:43,095 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:42:43,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,095 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:43,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,095 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:42:43,096 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:42:43,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,096 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:43,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,099 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,100 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,102 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:43,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:43,103 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:51,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:51,109 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:51,114 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:42:51,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:51,115 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 237, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:42:51,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:51,116 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 237, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:42:54,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:42:54,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:42:54,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.67 seconds 2025-02-15 12:42:54,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:54,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21545.98 MB 2025-02-15 12:42:54,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22384.71 MB 2025-02-15 12:42:54,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 838.73 MB 2025-02-15 12:42:54,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29840.38 MB 2025-02-15 12:42:54,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24165.48 MB 2025-02-15 12:42:54,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5674.89 MB 2025-02-15 12:42:54,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31244.65 MB 2025-02-15 12:42:54,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:42:54,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:42:54,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:42:54,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:54,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22384.71 MB 2025-02-15 12:42:54,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17573.79 MB 2025-02-15 12:42:54,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4810.92 MB 2025-02-15 12:42:54,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24165.48 MB 2025-02-15 12:42:54,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-15 12:42:54,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2298.48 MB 2025-02-15 12:42:54,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23510.07 MB 2025-02-15 12:42:55,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:42:55,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:42:55,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.10 seconds 2025-02-15 12:42:55,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:55,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17573.79 MB 2025-02-15 12:42:55,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17869.74 MB 2025-02-15 12:42:55,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 295.94 MB 2025-02-15 12:42:55,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21867.00 MB 2025-02-15 12:42:55,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19541.26 MB 2025-02-15 12:42:55,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2325.74 MB 2025-02-15 12:42:55,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21830.45 MB 2025-02-15 12:42:55,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:42:55,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:42:55,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:55,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:55,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17869.74 MB 2025-02-15 12:42:55,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18923.42 MB 2025-02-15 12:42:55,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1053.68 MB 2025-02-15 12:42:55,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19541.26 MB 2025-02-15 12:42:55,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21120.42 MB 2025-02-15 12:42:55,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1579.16 MB 2025-02-15 12:42:55,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19713.90 MB 2025-02-15 12:42:56,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:42:56,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:42:56,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:42:56,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18923.42 MB 2025-02-15 12:42:56,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20173.55 MB 2025-02-15 12:42:56,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1250.13 MB 2025-02-15 12:42:56,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21120.42 MB 2025-02-15 12:42:56,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24545.07 MB 2025-02-15 12:42:56,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3424.65 MB 2025-02-15 12:42:56,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23265.50 MB 2025-02-15 12:42:56,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:42:56,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:42:56,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:42:56,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17869.74 MB 2025-02-15 12:42:56,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20173.55 MB 2025-02-15 12:42:56,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2303.81 MB 2025-02-15 12:42:56,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19541.26 MB 2025-02-15 12:42:56,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24545.07 MB 2025-02-15 12:42:56,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5003.80 MB 2025-02-15 12:42:56,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23265.50 MB 2025-02-15 12:42:56,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:42:56,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:42:56,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:42:56,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20568.14 MB 2025-02-15 12:42:56,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20995.74 MB 2025-02-15 12:42:56,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 427.60 MB 2025-02-15 12:42:56,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24545.07 MB 2025-02-15 12:42:56,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24775.75 MB 2025-02-15 12:42:56,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 230.69 MB 2025-02-15 12:42:56,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21390.73 MB 2025-02-15 12:42:56,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:42:56,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:42:56,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:42:56,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21225.93 MB 2025-02-15 12:42:56,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21429.92 MB 2025-02-15 12:42:56,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.99 MB 2025-02-15 12:42:56,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24775.75 MB 2025-02-15 12:42:56,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24775.75 MB 2025-02-15 12:42:56,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:56,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21502.48 MB 2025-02-15 12:42:56,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:42:56,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:42:56,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.02 seconds 2025-02-15 12:42:56,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20720.26 MB 2025-02-15 12:42:56,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21630.77 MB 2025-02-15 12:42:56,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 910.51 MB 2025-02-15 12:42:56,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29840.38 MB 2025-02-15 12:42:56,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24775.75 MB 2025-02-15 12:42:56,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5064.62 MB 2025-02-15 12:42:56,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21630.77 MB 2025-02-15 12:42:56,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:42:56,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:42:56,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:42:56,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16293.89 MB 2025-02-15 12:42:56,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16394.16 MB 2025-02-15 12:42:56,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.27 MB 2025-02-15 12:42:56,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24775.75 MB 2025-02-15 12:42:56,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24775.75 MB 2025-02-15 12:42:56,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:42:56,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16995.78 MB 2025-02-15 12:42:56,419 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 12:42:56,419 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:42:56,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:42:56,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:42:56,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:42:56,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:42:56,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16394.16 MB 2025-02-15 12:42:56,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20580.44 MB 2025-02-15 12:42:56,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.28 MB 2025-02-15 12:42:56,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24775.75 MB 2025-02-15 12:42:56,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35244.74 MB 2025-02-15 12:42:56,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 12:42:56,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24766.36 MB 2025-02-15 12:42:56,586 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 12:42:56,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,587 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,588 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:42:56,593 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:42:56,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,594 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:42:56,594 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:42:56,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,594 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,595 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:56,601 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:42:56,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,601 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,602 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:56,602 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:42:56,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,602 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:56,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,603 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:42:56,603 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:42:56,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,603 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:42:56,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,606 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,607 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,608 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:42:56,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:42:56,609 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:11,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:11,014 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:11,019 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:43:11,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:11,020 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:43:11,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:11,021 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:43:13,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:43:13,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:43:13,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-15 12:43:13,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:13,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15970.65 MB 2025-02-15 12:43:13,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16515.65 MB 2025-02-15 12:43:13,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-15 12:43:13,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35244.74 MB 2025-02-15 12:43:13,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19014.88 MB 2025-02-15 12:43:13,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16229.86 MB 2025-02-15 12:43:13,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25442.83 MB 2025-02-15 12:43:13,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:43:13,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:43:13,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:13,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:13,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.65 MB 2025-02-15 12:43:13,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16744.58 MB 2025-02-15 12:43:13,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.93 MB 2025-02-15 12:43:13,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19014.88 MB 2025-02-15 12:43:13,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-15 12:43:13,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1056.96 MB 2025-02-15 12:43:13,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18608.57 MB 2025-02-15 12:43:14,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:43:14,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:43:14,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 12:43:14,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16744.58 MB 2025-02-15 12:43:14,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16942.32 MB 2025-02-15 12:43:14,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.74 MB 2025-02-15 12:43:14,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-15 12:43:14,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19543.36 MB 2025-02-15 12:43:14,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -528.48 MB 2025-02-15 12:43:14,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20915.27 MB 2025-02-15 12:43:14,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:43:14,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:43:14,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:43:14,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16942.25 MB 2025-02-15 12:43:14,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17645.93 MB 2025-02-15 12:43:14,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 703.68 MB 2025-02-15 12:43:14,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19543.36 MB 2025-02-15 12:43:14,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19543.36 MB 2025-02-15 12:43:14,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:14,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18173.93 MB 2025-02-15 12:43:14,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:43:14,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:43:14,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:43:14,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17645.93 MB 2025-02-15 12:43:14,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18481.85 MB 2025-02-15 12:43:14,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 12:43:14,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19543.36 MB 2025-02-15 12:43:14,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21657.29 MB 2025-02-15 12:43:14,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2113.93 MB 2025-02-15 12:43:14,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20550.20 MB 2025-02-15 12:43:14,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:43:14,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:43:14,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:43:14,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16942.25 MB 2025-02-15 12:43:14,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18481.85 MB 2025-02-15 12:43:14,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1539.60 MB 2025-02-15 12:43:14,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19543.36 MB 2025-02-15 12:43:14,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21657.29 MB 2025-02-15 12:43:14,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2113.93 MB 2025-02-15 12:43:14,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20550.20 MB 2025-02-15 12:43:14,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:43:14,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:43:14,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:43:14,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18745.50 MB 2025-02-15 12:43:14,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19031.21 MB 2025-02-15 12:43:14,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.71 MB 2025-02-15 12:43:14,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21657.29 MB 2025-02-15 12:43:14,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 12:43:14,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 12:43:14,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19305.70 MB 2025-02-15 12:43:14,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:43:14,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:43:14,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:14,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19185.02 MB 2025-02-15 12:43:14,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19389.34 MB 2025-02-15 12:43:14,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.32 MB 2025-02-15 12:43:14,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-15 12:43:14,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 12:43:14,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:14,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19400.43 MB 2025-02-15 12:43:14,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:43:14,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:43:14,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.30 seconds 2025-02-15 12:43:14,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15433.91 MB 2025-02-15 12:43:14,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19590.39 MB 2025-02-15 12:43:14,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4156.48 MB 2025-02-15 12:43:14,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35244.74 MB 2025-02-15 12:43:14,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 12:43:14,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13434.36 MB 2025-02-15 12:43:14,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19590.39 MB 2025-02-15 12:43:14,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:43:14,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:43:14,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:43:14,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19590.39 MB 2025-02-15 12:43:14,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19690.84 MB 2025-02-15 12:43:14,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.45 MB 2025-02-15 12:43:14,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-15 12:43:14,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-15 12:43:14,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:14,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20293.57 MB 2025-02-15 12:43:14,601 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 12:43:14,601 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:43:14,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:43:14,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:43:14,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:43:14,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:14,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16030.45 MB 2025-02-15 12:43:14,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20224.76 MB 2025-02-15 12:43:14,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-15 12:43:14,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-15 12:43:14,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32296.14 MB 2025-02-15 12:43:14,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 12:43:14,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24419.06 MB 2025-02-15 12:43:14,763 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 12:43:14,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:43:14,770 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:43:14,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,771 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:43:14,771 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:43:14,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,772 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,772 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:14,778 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:43:14,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,779 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,779 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:14,779 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:43:14,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,780 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:14,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,780 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:43:14,780 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:43:14,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,781 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:14,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,783 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,784 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,785 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:14,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:14,786 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:25,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:25,381 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:25,385 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:43:25,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:25,386 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 375, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:43:25,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:25,387 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 375, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:43:31,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:43:31,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:43:31,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.79 seconds 2025-02-15 12:43:31,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:31,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17632.15 MB 2025-02-15 12:43:31,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18959.25 MB 2025-02-15 12:43:31,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1327.10 MB 2025-02-15 12:43:31,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32296.14 MB 2025-02-15 12:43:31,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21871.20 MB 2025-02-15 12:43:31,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10424.94 MB 2025-02-15 12:43:31,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27783.80 MB 2025-02-15 12:43:31,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:43:31,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:43:31,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:43:31,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:31,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18959.25 MB 2025-02-15 12:43:31,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19259.01 MB 2025-02-15 12:43:31,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.76 MB 2025-02-15 12:43:31,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21871.20 MB 2025-02-15 12:43:31,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26459.77 MB 2025-02-15 12:43:31,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4588.57 MB 2025-02-15 12:43:31,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23540.21 MB 2025-02-15 12:43:32,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:43:32,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:43:32,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.57 seconds 2025-02-15 12:43:32,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:32,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19259.01 MB 2025-02-15 12:43:32,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19691.65 MB 2025-02-15 12:43:32,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.64 MB 2025-02-15 12:43:32,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26459.77 MB 2025-02-15 12:43:32,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22758.29 MB 2025-02-15 12:43:32,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3701.47 MB 2025-02-15 12:43:32,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23684.50 MB 2025-02-15 12:43:32,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:43:32,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:43:32,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:32,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:32,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19691.65 MB 2025-02-15 12:43:32,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21232.79 MB 2025-02-15 12:43:32,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1541.14 MB 2025-02-15 12:43:32,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22758.29 MB 2025-02-15 12:43:32,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25067.26 MB 2025-02-15 12:43:32,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2308.96 MB 2025-02-15 12:43:32,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22388.52 MB 2025-02-15 12:43:32,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:43:32,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:43:32,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:43:32,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:32,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.79 MB 2025-02-15 12:43:32,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23060.97 MB 2025-02-15 12:43:32,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1828.17 MB 2025-02-15 12:43:32,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25067.26 MB 2025-02-15 12:43:32,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29685.19 MB 2025-02-15 12:43:32,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4617.93 MB 2025-02-15 12:43:32,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27581.64 MB 2025-02-15 12:43:32,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:43:32,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:43:32,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:43:32,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:32,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19691.65 MB 2025-02-15 12:43:32,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23060.97 MB 2025-02-15 12:43:32,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3369.32 MB 2025-02-15 12:43:32,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22758.29 MB 2025-02-15 12:43:32,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29685.19 MB 2025-02-15 12:43:32,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6926.89 MB 2025-02-15 12:43:32,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27581.64 MB 2025-02-15 12:43:33,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:43:33,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:43:33,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:43:33,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:33,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23637.81 MB 2025-02-15 12:43:33,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24262.92 MB 2025-02-15 12:43:33,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.11 MB 2025-02-15 12:43:33,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29685.19 MB 2025-02-15 12:43:33,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30024.93 MB 2025-02-15 12:43:33,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 339.74 MB 2025-02-15 12:43:33,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24839.77 MB 2025-02-15 12:43:33,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:43:33,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:43:33,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:33,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:33,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24599.43 MB 2025-02-15 12:43:33,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24806.14 MB 2025-02-15 12:43:33,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.71 MB 2025-02-15 12:43:33,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30024.93 MB 2025-02-15 12:43:33,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30024.93 MB 2025-02-15 12:43:33,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:33,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24929.22 MB 2025-02-15 12:43:33,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:43:33,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:43:33,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.71 seconds 2025-02-15 12:43:33,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:33,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16325.62 MB 2025-02-15 12:43:33,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25007.21 MB 2025-02-15 12:43:33,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8681.60 MB 2025-02-15 12:43:33,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32296.14 MB 2025-02-15 12:43:33,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30024.93 MB 2025-02-15 12:43:33,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2271.22 MB 2025-02-15 12:43:33,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25007.21 MB 2025-02-15 12:43:33,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:43:33,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:43:33,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:43:33,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:33,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25007.21 MB 2025-02-15 12:43:33,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25107.68 MB 2025-02-15 12:43:33,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:43:33,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30024.93 MB 2025-02-15 12:43:33,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30024.93 MB 2025-02-15 12:43:33,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:33,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25710.48 MB 2025-02-15 12:43:33,384 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:43:33,384 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:43:33,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:43:33,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:43:33,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:43:33,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:33,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.68 MB 2025-02-15 12:43:33,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21586.57 MB 2025-02-15 12:43:33,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3521.11 MB 2025-02-15 12:43:33,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30024.93 MB 2025-02-15 12:43:33,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40514.88 MB 2025-02-15 12:43:33,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:43:33,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25780.87 MB 2025-02-15 12:43:33,551 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:43:33,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,552 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,553 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:43:33,558 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:43:33,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,559 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:43:33,559 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:43:33,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,560 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,560 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:33,566 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:43:33,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,567 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,567 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:33,567 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:43:33,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,568 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:33,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,568 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:43:33,568 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:43:33,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,569 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:33,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,572 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,572 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,573 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:33,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:33,575 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:40,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:40,709 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:40,714 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:43:40,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:40,715 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:43:40,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:40,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:43:43,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:43:43,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:43:43,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-15 12:43:43,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:43,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16436.89 MB 2025-02-15 12:43:43,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17095.13 MB 2025-02-15 12:43:43,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-15 12:43:43,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40514.88 MB 2025-02-15 12:43:43,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21718.11 MB 2025-02-15 12:43:43,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18796.77 MB 2025-02-15 12:43:43,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.26 MB 2025-02-15 12:43:43,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:43:43,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:43:43,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:43,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:43,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17095.13 MB 2025-02-15 12:43:43,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17413.98 MB 2025-02-15 12:43:43,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.85 MB 2025-02-15 12:43:43,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21718.11 MB 2025-02-15 12:43:43,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21718.11 MB 2025-02-15 12:43:43,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:43,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19707.70 MB 2025-02-15 12:43:44,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:43:44,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:43:44,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-15 12:43:44,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17413.98 MB 2025-02-15 12:43:44,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17660.83 MB 2025-02-15 12:43:44,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-15 12:43:44,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21718.11 MB 2025-02-15 12:43:44,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20445.13 MB 2025-02-15 12:43:44,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1272.97 MB 2025-02-15 12:43:44,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21584.67 MB 2025-02-15 12:43:44,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:43:44,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:43:44,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:44,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17660.83 MB 2025-02-15 12:43:44,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18539.25 MB 2025-02-15 12:43:44,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-15 12:43:44,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20445.13 MB 2025-02-15 12:43:44,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20887.63 MB 2025-02-15 12:43:44,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 442.50 MB 2025-02-15 12:43:44,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19198.35 MB 2025-02-15 12:43:44,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:43:44,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:43:44,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:43:44,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18539.25 MB 2025-02-15 12:43:44,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19581.74 MB 2025-02-15 12:43:44,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-15 12:43:44,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20887.63 MB 2025-02-15 12:43:44,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23750.25 MB 2025-02-15 12:43:44,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2862.61 MB 2025-02-15 12:43:44,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22160.59 MB 2025-02-15 12:43:44,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:43:44,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:43:44,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:43:44,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17660.83 MB 2025-02-15 12:43:44,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19581.74 MB 2025-02-15 12:43:44,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-15 12:43:44,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20445.13 MB 2025-02-15 12:43:44,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23750.25 MB 2025-02-15 12:43:44,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 12:43:44,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22160.59 MB 2025-02-15 12:43:44,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:43:44,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:43:44,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:43:44,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19910.87 MB 2025-02-15 12:43:44,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20268.31 MB 2025-02-15 12:43:44,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 357.44 MB 2025-02-15 12:43:44,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23750.25 MB 2025-02-15 12:43:44,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23943.18 MB 2025-02-15 12:43:44,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 192.94 MB 2025-02-15 12:43:44,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20602.24 MB 2025-02-15 12:43:44,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:43:44,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:43:44,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:44,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20460.31 MB 2025-02-15 12:43:44,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20661.75 MB 2025-02-15 12:43:44,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.44 MB 2025-02-15 12:43:44,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23943.18 MB 2025-02-15 12:43:44,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23947.38 MB 2025-02-15 12:43:44,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:43:44,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20698.72 MB 2025-02-15 12:43:44,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:43:44,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:43:44,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.00 seconds 2025-02-15 12:43:44,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15788.85 MB 2025-02-15 12:43:44,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20862.82 MB 2025-02-15 12:43:44,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5073.97 MB 2025-02-15 12:43:44,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40514.88 MB 2025-02-15 12:43:44,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23947.38 MB 2025-02-15 12:43:44,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16567.50 MB 2025-02-15 12:43:44,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20862.82 MB 2025-02-15 12:43:44,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:43:44,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:43:44,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:43:44,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:44,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20862.82 MB 2025-02-15 12:43:44,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20963.29 MB 2025-02-15 12:43:44,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:43:44,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23947.38 MB 2025-02-15 12:43:44,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23947.38 MB 2025-02-15 12:43:44,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:44,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21566.09 MB 2025-02-15 12:43:44,998 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:43:44,998 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:43:45,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:43:45,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:43:45,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:43:45,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:45,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16484.43 MB 2025-02-15 12:43:45,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20678.92 MB 2025-02-15 12:43:45,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:43:45,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23947.38 MB 2025-02-15 12:43:45,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34437.33 MB 2025-02-15 12:43:45,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:43:45,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24873.22 MB 2025-02-15 12:43:45,162 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:43:45,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:43:45,168 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:43:45,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,169 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:43:45,170 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:43:45,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,170 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,171 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:45,177 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:43:45,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,177 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,178 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:45,178 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:43:45,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,178 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:45,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,179 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:43:45,179 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:43:45,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,179 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:45,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,182 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,183 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,183 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,184 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:45,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:45,186 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:54,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:54,003 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:54,007 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:43:54,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:54,009 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:43:54,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:54,009 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:43:56,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:43:56,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:43:56,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.39 seconds 2025-02-15 12:43:56,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:56,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16328.67 MB 2025-02-15 12:43:56,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16870.13 MB 2025-02-15 12:43:56,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 12:43:56,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34437.33 MB 2025-02-15 12:43:56,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20170.41 MB 2025-02-15 12:43:56,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14266.93 MB 2025-02-15 12:43:56,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25800.84 MB 2025-02-15 12:43:56,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:43:56,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:43:56,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:56,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:56,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16870.13 MB 2025-02-15 12:43:56,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17090.32 MB 2025-02-15 12:43:56,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-15 12:43:56,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20170.41 MB 2025-02-15 12:43:56,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21208.50 MB 2025-02-15 12:43:56,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1038.09 MB 2025-02-15 12:43:56,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18934.96 MB 2025-02-15 12:43:57,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:43:57,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:43:57,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.73 seconds 2025-02-15 12:43:57,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.32 MB 2025-02-15 12:43:57,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17285.41 MB 2025-02-15 12:43:57,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-15 12:43:57,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21208.50 MB 2025-02-15 12:43:57,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20768.10 MB 2025-02-15 12:43:57,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -440.40 MB 2025-02-15 12:43:57,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21261.01 MB 2025-02-15 12:43:57,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:43:57,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:43:57,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:43:57,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17285.34 MB 2025-02-15 12:43:57,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17979.58 MB 2025-02-15 12:43:57,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-15 12:43:57,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20768.10 MB 2025-02-15 12:43:57,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20768.10 MB 2025-02-15 12:43:57,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:57,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18500.49 MB 2025-02-15 12:43:57,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:43:57,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:43:57,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:43:57,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17979.58 MB 2025-02-15 12:43:57,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18803.50 MB 2025-02-15 12:43:57,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-15 12:43:57,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20768.10 MB 2025-02-15 12:43:57,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22160.61 MB 2025-02-15 12:43:57,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 12:43:57,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20840.98 MB 2025-02-15 12:43:57,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:43:57,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:43:57,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:43:57,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17285.34 MB 2025-02-15 12:43:57,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18803.50 MB 2025-02-15 12:43:57,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-15 12:43:57,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20768.10 MB 2025-02-15 12:43:57,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22160.61 MB 2025-02-15 12:43:57,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 12:43:57,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20840.98 MB 2025-02-15 12:43:57,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:43:57,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:43:57,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:43:57,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19063.61 MB 2025-02-15 12:43:57,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19345.49 MB 2025-02-15 12:43:57,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-15 12:43:57,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22160.61 MB 2025-02-15 12:43:57,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22313.70 MB 2025-02-15 12:43:57,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-15 12:43:57,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19618.87 MB 2025-02-15 12:43:57,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:43:57,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:43:57,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:43:57,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19497.23 MB 2025-02-15 12:43:57,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19698.90 MB 2025-02-15 12:43:57,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.67 MB 2025-02-15 12:43:57,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22313.70 MB 2025-02-15 12:43:57,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22315.79 MB 2025-02-15 12:43:57,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 12:43:57,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19709.21 MB 2025-02-15 12:43:57,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:43:57,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:43:57,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-15 12:43:57,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15795.60 MB 2025-02-15 12:43:57,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19899.88 MB 2025-02-15 12:43:57,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4104.27 MB 2025-02-15 12:43:57,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34437.33 MB 2025-02-15 12:43:57,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22315.79 MB 2025-02-15 12:43:57,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12121.54 MB 2025-02-15 12:43:57,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19899.88 MB 2025-02-15 12:43:57,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:43:57,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:43:57,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:43:57,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19899.88 MB 2025-02-15 12:43:57,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20000.29 MB 2025-02-15 12:43:57,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.42 MB 2025-02-15 12:43:57,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22315.79 MB 2025-02-15 12:43:57,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22315.79 MB 2025-02-15 12:43:57,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:43:57,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20602.80 MB 2025-02-15 12:43:57,582 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 12:43:57,582 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:43:57,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:43:57,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:43:57,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:43:57,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:43:57,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20000.29 MB 2025-02-15 12:43:57,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24192.73 MB 2025-02-15 12:43:57,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.43 MB 2025-02-15 12:43:57,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22315.79 MB 2025-02-15 12:43:57,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32799.46 MB 2025-02-15 12:43:57,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-15 12:43:57,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28384.93 MB 2025-02-15 12:43:57,746 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 12:43:57,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,748 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,749 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:43:57,753 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:43:57,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,754 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:43:57,754 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:43:57,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,755 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,755 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:57,761 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:43:57,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,762 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,762 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:57,762 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:43:57,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,763 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:57,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,763 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:43:57,763 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:43:57,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,764 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:43:57,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,767 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,768 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,769 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:43:57,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:43:57,772 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:12,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:12,694 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:12,699 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:44:12,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:12,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 166, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:44:12,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:12,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 166, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:44:15,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:44:15,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:44:15,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-15 12:44:15,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:15,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25593.05 MB 2025-02-15 12:44:15,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26180.51 MB 2025-02-15 12:44:15,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 587.46 MB 2025-02-15 12:44:15,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-15 12:44:15,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28607.25 MB 2025-02-15 12:44:15,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4192.21 MB 2025-02-15 12:44:15,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35065.65 MB 2025-02-15 12:44:15,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:44:15,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:44:15,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:15,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:15,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26180.51 MB 2025-02-15 12:44:15,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26339.25 MB 2025-02-15 12:44:15,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 158.73 MB 2025-02-15 12:44:15,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28607.25 MB 2025-02-15 12:44:15,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29911.68 MB 2025-02-15 12:44:15,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1304.43 MB 2025-02-15 12:44:15,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28260.44 MB 2025-02-15 12:44:16,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:44:16,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:44:16,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-15 12:44:16,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26339.25 MB 2025-02-15 12:44:16,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26535.66 MB 2025-02-15 12:44:16,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-15 12:44:16,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29911.68 MB 2025-02-15 12:44:16,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -872.42 MB 2025-02-15 12:44:16,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30509.94 MB 2025-02-15 12:44:16,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:44:16,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:44:16,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:44:16,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26535.59 MB 2025-02-15 12:44:16,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.55 MB 2025-02-15 12:44:16,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-15 12:44:16,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27759.01 MB 2025-02-15 12:44:16,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:44:16,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:44:16,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:44:16,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27234.55 MB 2025-02-15 12:44:16,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19012.01 MB 2025-02-15 12:44:16,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8222.54 MB 2025-02-15 12:44:16,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27570.79 MB 2025-02-15 12:44:16,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:44:16,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:44:16,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:44:16,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26535.59 MB 2025-02-15 12:44:16,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19012.01 MB 2025-02-15 12:44:16,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7523.58 MB 2025-02-15 12:44:16,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27570.79 MB 2025-02-15 12:44:16,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:44:16,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:44:16,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:44:16,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19274.15 MB 2025-02-15 12:44:16,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19557.68 MB 2025-02-15 12:44:16,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.53 MB 2025-02-15 12:44:16,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19831.39 MB 2025-02-15 12:44:16,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:44:16,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:44:16,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:16,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19710.46 MB 2025-02-15 12:44:16,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19913.33 MB 2025-02-15 12:44:16,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 202.87 MB 2025-02-15 12:44:16,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19925.45 MB 2025-02-15 12:44:16,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:44:16,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:44:16,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-15 12:44:16,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25014.69 MB 2025-02-15 12:44:16,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20114.31 MB 2025-02-15 12:44:16,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4900.38 MB 2025-02-15 12:44:16,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-15 12:44:16,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3760.19 MB 2025-02-15 12:44:16,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20114.31 MB 2025-02-15 12:44:16,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:44:16,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:44:16,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:44:16,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20114.31 MB 2025-02-15 12:44:16,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20214.72 MB 2025-02-15 12:44:16,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.42 MB 2025-02-15 12:44:16,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29039.26 MB 2025-02-15 12:44:16,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:16,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20817.23 MB 2025-02-15 12:44:16,445 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 12:44:16,446 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:44:16,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:44:16,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:44:16,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:44:16,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:16,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16556.44 MB 2025-02-15 12:44:16,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20748.87 MB 2025-02-15 12:44:16,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.43 MB 2025-02-15 12:44:16,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29039.26 MB 2025-02-15 12:44:16,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33233.57 MB 2025-02-15 12:44:16,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 12:44:16,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24940.79 MB 2025-02-15 12:44:16,610 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 12:44:16,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,612 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,613 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:44:16,617 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:44:16,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,618 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:44:16,618 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:44:16,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,619 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,620 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:16,625 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:44:16,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,626 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,626 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:16,626 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:44:16,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,627 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:16,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,627 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:44:16,627 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:44:16,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,628 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:16,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,632 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,633 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,635 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:16,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:16,637 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:24,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:24,185 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:24,189 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:44:24,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:24,191 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:44:24,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:24,192 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:44:27,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:44:27,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:44:27,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.71 seconds 2025-02-15 12:44:27,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:27,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22657.87 MB 2025-02-15 12:44:27,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23503.68 MB 2025-02-15 12:44:27,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.81 MB 2025-02-15 12:44:27,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33233.57 MB 2025-02-15 12:44:27,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25937.58 MB 2025-02-15 12:44:27,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7295.99 MB 2025-02-15 12:44:27,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32356.54 MB 2025-02-15 12:44:27,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:44:27,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:44:27,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:44:27,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:27,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23503.68 MB 2025-02-15 12:44:27,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23914.12 MB 2025-02-15 12:44:27,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.45 MB 2025-02-15 12:44:27,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25937.58 MB 2025-02-15 12:44:27,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28877.78 MB 2025-02-15 12:44:27,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2940.21 MB 2025-02-15 12:44:27,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26861.40 MB 2025-02-15 12:44:29,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:44:29,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:44:29,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.16 seconds 2025-02-15 12:44:29,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23914.12 MB 2025-02-15 12:44:29,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24231.30 MB 2025-02-15 12:44:29,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.18 MB 2025-02-15 12:44:29,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28877.78 MB 2025-02-15 12:44:29,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26415.73 MB 2025-02-15 12:44:29,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2462.06 MB 2025-02-15 12:44:29,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28169.75 MB 2025-02-15 12:44:29,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:44:29,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:44:29,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:29,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24231.30 MB 2025-02-15 12:44:29,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25360.55 MB 2025-02-15 12:44:29,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1129.25 MB 2025-02-15 12:44:29,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26415.73 MB 2025-02-15 12:44:29,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28110.23 MB 2025-02-15 12:44:29,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1694.50 MB 2025-02-15 12:44:29,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26207.47 MB 2025-02-15 12:44:29,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:44:29,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:44:29,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:44:29,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25360.55 MB 2025-02-15 12:44:29,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26700.35 MB 2025-02-15 12:44:29,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1339.80 MB 2025-02-15 12:44:29,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28110.23 MB 2025-02-15 12:44:29,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31497.13 MB 2025-02-15 12:44:29,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3386.90 MB 2025-02-15 12:44:29,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30014.08 MB 2025-02-15 12:44:29,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:44:29,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:44:29,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:44:29,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24231.30 MB 2025-02-15 12:44:29,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26700.35 MB 2025-02-15 12:44:29,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2469.04 MB 2025-02-15 12:44:29,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26415.73 MB 2025-02-15 12:44:29,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31497.13 MB 2025-02-15 12:44:29,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5081.40 MB 2025-02-15 12:44:29,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30014.08 MB 2025-02-15 12:44:29,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:44:29,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:44:29,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:44:29,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27123.25 MB 2025-02-15 12:44:29,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27581.53 MB 2025-02-15 12:44:29,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.28 MB 2025-02-15 12:44:29,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31497.13 MB 2025-02-15 12:44:29,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31746.69 MB 2025-02-15 12:44:29,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 249.56 MB 2025-02-15 12:44:29,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28004.44 MB 2025-02-15 12:44:29,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:44:29,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:44:29,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:29,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27828.24 MB 2025-02-15 12:44:29,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28034.04 MB 2025-02-15 12:44:29,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.81 MB 2025-02-15 12:44:29,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31746.69 MB 2025-02-15 12:44:29,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31746.69 MB 2025-02-15 12:44:29,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:29,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28118.77 MB 2025-02-15 12:44:29,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:44:29,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:44:29,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.21 seconds 2025-02-15 12:44:29,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21825.17 MB 2025-02-15 12:44:29,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28234.85 MB 2025-02-15 12:44:29,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6409.67 MB 2025-02-15 12:44:29,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33233.57 MB 2025-02-15 12:44:29,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31746.69 MB 2025-02-15 12:44:29,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1486.88 MB 2025-02-15 12:44:29,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28234.85 MB 2025-02-15 12:44:29,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:44:29,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:44:29,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:44:29,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28234.85 MB 2025-02-15 12:44:29,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28335.18 MB 2025-02-15 12:44:29,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.33 MB 2025-02-15 12:44:29,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31746.69 MB 2025-02-15 12:44:29,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31746.69 MB 2025-02-15 12:44:29,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:29,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28937.17 MB 2025-02-15 12:44:29,702 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 12:44:29,703 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:44:29,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:44:29,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:44:29,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:44:29,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:29,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28335.18 MB 2025-02-15 12:44:29,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26849.25 MB 2025-02-15 12:44:29,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1485.93 MB 2025-02-15 12:44:29,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31746.69 MB 2025-02-15 12:44:29,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42221.96 MB 2025-02-15 12:44:29,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 12:44:29,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31037.58 MB 2025-02-15 12:44:29,959 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 12:44:29,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:29,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,964 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:44:29,971 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:44:29,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:44:29,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:44:29,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,974 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:29,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,976 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:29,985 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:44:29,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,986 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:29,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,987 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:29,987 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:44:29,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,988 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:29,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,989 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:44:29,989 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:44:29,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:29,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:30,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:30,000 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:30,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:30,003 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:30,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:30,006 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:30,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:30,010 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:38,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:38,084 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:38,089 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:44:38,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:38,090 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:44:38,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:38,091 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:44:40,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:44:40,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:44:40,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-15 12:44:40,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:40,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16728.69 MB 2025-02-15 12:44:40,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17287.84 MB 2025-02-15 12:44:40,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-15 12:44:40,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42221.96 MB 2025-02-15 12:44:40,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21879.59 MB 2025-02-15 12:44:40,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20342.37 MB 2025-02-15 12:44:40,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26200.06 MB 2025-02-15 12:44:40,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:44:40,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:44:40,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:40,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:40,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17287.84 MB 2025-02-15 12:44:40,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17558.75 MB 2025-02-15 12:44:40,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.91 MB 2025-02-15 12:44:40,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21879.59 MB 2025-02-15 12:44:40,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21879.59 MB 2025-02-15 12:44:40,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:40,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19507.18 MB 2025-02-15 12:44:41,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:44:41,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:44:41,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-15 12:44:41,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17558.75 MB 2025-02-15 12:44:41,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17768.43 MB 2025-02-15 12:44:41,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.68 MB 2025-02-15 12:44:41,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21879.59 MB 2025-02-15 12:44:41,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21596.47 MB 2025-02-15 12:44:41,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -283.12 MB 2025-02-15 12:44:41,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21729.44 MB 2025-02-15 12:44:41,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:44:41,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:44:41,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:44:41,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17768.36 MB 2025-02-15 12:44:41,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18514.55 MB 2025-02-15 12:44:41,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.18 MB 2025-02-15 12:44:41,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21596.47 MB 2025-02-15 12:44:41,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21596.47 MB 2025-02-15 12:44:41,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:41,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19074.44 MB 2025-02-15 12:44:41,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:44:41,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:44:41,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:44:41,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18514.55 MB 2025-02-15 12:44:41,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19400.47 MB 2025-02-15 12:44:41,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 885.92 MB 2025-02-15 12:44:41,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21596.47 MB 2025-02-15 12:44:41,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23462.94 MB 2025-02-15 12:44:41,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1866.47 MB 2025-02-15 12:44:41,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21592.52 MB 2025-02-15 12:44:41,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:44:41,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:44:41,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:44:41,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17768.36 MB 2025-02-15 12:44:41,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19400.47 MB 2025-02-15 12:44:41,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1632.10 MB 2025-02-15 12:44:41,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21596.47 MB 2025-02-15 12:44:41,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23462.94 MB 2025-02-15 12:44:41,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1866.47 MB 2025-02-15 12:44:41,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21592.52 MB 2025-02-15 12:44:41,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:44:41,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:44:41,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:44:41,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19680.04 MB 2025-02-15 12:44:41,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19983.01 MB 2025-02-15 12:44:41,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.97 MB 2025-02-15 12:44:41,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23462.94 MB 2025-02-15 12:44:41,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23628.61 MB 2025-02-15 12:44:41,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 12:44:41,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20270.69 MB 2025-02-15 12:44:41,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:44:41,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:44:41,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:41,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20146.11 MB 2025-02-15 12:44:41,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20347.33 MB 2025-02-15 12:44:41,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.22 MB 2025-02-15 12:44:41,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23628.61 MB 2025-02-15 12:44:41,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23628.61 MB 2025-02-15 12:44:41,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:41,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20366.54 MB 2025-02-15 12:44:41,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:44:41,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:44:41,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.41 seconds 2025-02-15 12:44:41,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16178.20 MB 2025-02-15 12:44:41,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20548.40 MB 2025-02-15 12:44:41,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4370.20 MB 2025-02-15 12:44:41,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42221.96 MB 2025-02-15 12:44:41,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23628.61 MB 2025-02-15 12:44:41,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18593.35 MB 2025-02-15 12:44:41,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20548.40 MB 2025-02-15 12:44:41,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:44:41,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:44:41,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:44:41,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20548.40 MB 2025-02-15 12:44:41,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20648.87 MB 2025-02-15 12:44:41,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:44:41,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23628.61 MB 2025-02-15 12:44:41,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23628.61 MB 2025-02-15 12:44:41,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:41,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21251.67 MB 2025-02-15 12:44:41,778 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:44:41,778 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:44:41,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:44:41,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:44:41,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:44:41,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:41,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16798.66 MB 2025-02-15 12:44:41,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20993.15 MB 2025-02-15 12:44:41,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:44:41,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23628.61 MB 2025-02-15 12:44:41,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34118.57 MB 2025-02-15 12:44:41,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:44:41,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25187.45 MB 2025-02-15 12:44:41,945 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:44:41,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,946 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,947 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:44:41,952 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:44:41,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,953 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:44:41,953 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:44:41,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,954 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,954 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:41,960 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:44:41,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,961 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,961 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:41,961 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:44:41,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,962 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:41,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,962 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:44:41,962 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:44:41,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,963 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:41,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,965 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,966 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,967 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:41,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:41,969 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:50,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:50,745 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:50,750 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:44:50,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:50,751 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:44:50,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:50,751 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:44:53,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:44:53,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:44:53,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.71 seconds 2025-02-15 12:44:53,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:53,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22456.18 MB 2025-02-15 12:44:53,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23075.50 MB 2025-02-15 12:44:53,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.32 MB 2025-02-15 12:44:53,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34118.57 MB 2025-02-15 12:44:53,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26633.83 MB 2025-02-15 12:44:53,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7484.74 MB 2025-02-15 12:44:53,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31928.36 MB 2025-02-15 12:44:53,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:44:53,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:44:53,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:53,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:53,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23075.50 MB 2025-02-15 12:44:53,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23375.49 MB 2025-02-15 12:44:53,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.99 MB 2025-02-15 12:44:53,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26633.83 MB 2025-02-15 12:44:53,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27254.59 MB 2025-02-15 12:44:53,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 620.76 MB 2025-02-15 12:44:53,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25533.56 MB 2025-02-15 12:44:54,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:44:54,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:44:54,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 12:44:54,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23375.49 MB 2025-02-15 12:44:54,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23607.73 MB 2025-02-15 12:44:54,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.24 MB 2025-02-15 12:44:54,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27254.59 MB 2025-02-15 12:44:54,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27283.95 MB 2025-02-15 12:44:54,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29.36 MB 2025-02-15 12:44:54,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27546.18 MB 2025-02-15 12:44:54,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:44:54,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:44:54,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:54,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23607.73 MB 2025-02-15 12:44:54,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24434.20 MB 2025-02-15 12:44:54,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 826.47 MB 2025-02-15 12:44:54,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-15 12:44:54,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27283.95 MB 2025-02-15 12:44:54,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:54,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25054.33 MB 2025-02-15 12:44:54,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:44:54,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:44:54,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:44:54,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24434.20 MB 2025-02-15 12:44:54,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25415.94 MB 2025-02-15 12:44:54,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 981.73 MB 2025-02-15 12:44:54,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-15 12:44:54,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29351.74 MB 2025-02-15 12:44:54,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2067.79 MB 2025-02-15 12:44:54,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27842.31 MB 2025-02-15 12:44:54,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:44:54,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:44:54,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:44:54,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23607.73 MB 2025-02-15 12:44:54,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25415.94 MB 2025-02-15 12:44:54,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1808.20 MB 2025-02-15 12:44:54,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-15 12:44:54,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29351.74 MB 2025-02-15 12:44:54,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2067.79 MB 2025-02-15 12:44:54,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27842.31 MB 2025-02-15 12:44:54,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:44:54,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:44:54,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:44:54,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25725.59 MB 2025-02-15 12:44:54,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26061.16 MB 2025-02-15 12:44:54,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 335.56 MB 2025-02-15 12:44:54,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29351.74 MB 2025-02-15 12:44:54,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29534.19 MB 2025-02-15 12:44:54,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 12:44:54,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26379.37 MB 2025-02-15 12:44:54,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:44:54,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:44:54,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:44:54,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26241.80 MB 2025-02-15 12:44:54,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26447.49 MB 2025-02-15 12:44:54,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.69 MB 2025-02-15 12:44:54,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29534.19 MB 2025-02-15 12:44:54,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29534.19 MB 2025-02-15 12:44:54,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:54,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26470.26 MB 2025-02-15 12:44:54,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:44:54,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:44:54,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.75 seconds 2025-02-15 12:44:54,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21846.47 MB 2025-02-15 12:44:54,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26648.12 MB 2025-02-15 12:44:54,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4801.66 MB 2025-02-15 12:44:54,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34118.57 MB 2025-02-15 12:44:54,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29534.19 MB 2025-02-15 12:44:54,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4584.37 MB 2025-02-15 12:44:54,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26648.12 MB 2025-02-15 12:44:54,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:44:54,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:44:54,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:44:54,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26648.12 MB 2025-02-15 12:44:54,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26748.37 MB 2025-02-15 12:44:54,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.25 MB 2025-02-15 12:44:54,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29534.19 MB 2025-02-15 12:44:54,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29534.19 MB 2025-02-15 12:44:54,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:44:54,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27349.84 MB 2025-02-15 12:44:54,784 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 12:44:54,784 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:44:54,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:44:54,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:44:54,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:44:54,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:44:54,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22511.62 MB 2025-02-15 12:44:54,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26697.54 MB 2025-02-15 12:44:54,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-15 12:44:54,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29534.19 MB 2025-02-15 12:44:54,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39998.98 MB 2025-02-15 12:44:54,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 12:44:54,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30882.28 MB 2025-02-15 12:44:54,947 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 12:44:54,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,949 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,949 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:44:54,954 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:44:54,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,955 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:44:54,955 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:44:54,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,956 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,956 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:54,962 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:44:54,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,963 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,963 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:54,963 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:44:54,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,964 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:54,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,964 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:44:54,964 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:44:54,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,965 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:44:54,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,968 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,970 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,971 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:44:54,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:44:54,974 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:03,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:03,048 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:03,053 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:45:03,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:03,054 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 187, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:45:03,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:03,055 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 187, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:45:05,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:45:05,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:45:05,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-15 12:45:05,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:05,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22661.53 MB 2025-02-15 12:45:05,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23323.31 MB 2025-02-15 12:45:05,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.78 MB 2025-02-15 12:45:05,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39998.98 MB 2025-02-15 12:45:05,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26254.25 MB 2025-02-15 12:45:05,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13744.73 MB 2025-02-15 12:45:05,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32133.71 MB 2025-02-15 12:45:05,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:45:05,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:45:05,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:05,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:05,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23323.31 MB 2025-02-15 12:45:05,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23645.03 MB 2025-02-15 12:45:05,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 321.72 MB 2025-02-15 12:45:05,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26254.25 MB 2025-02-15 12:45:05,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27900.51 MB 2025-02-15 12:45:05,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1646.26 MB 2025-02-15 12:45:05,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25951.99 MB 2025-02-15 12:45:06,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:45:06,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:45:06,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-15 12:45:06,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:06,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23645.03 MB 2025-02-15 12:45:06,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23893.19 MB 2025-02-15 12:45:06,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.17 MB 2025-02-15 12:45:06,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-15 12:45:06,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27237.81 MB 2025-02-15 12:45:06,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -662.70 MB 2025-02-15 12:45:06,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27815.71 MB 2025-02-15 12:45:06,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:45:06,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:45:06,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:06,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:06,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23893.19 MB 2025-02-15 12:45:06,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24776.34 MB 2025-02-15 12:45:06,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 883.14 MB 2025-02-15 12:45:06,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27237.81 MB 2025-02-15 12:45:06,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27237.81 MB 2025-02-15 12:45:06,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:06,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25438.99 MB 2025-02-15 12:45:06,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:45:06,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:45:06,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:45:06,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:06,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24776.34 MB 2025-02-15 12:45:06,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20337.13 MB 2025-02-15 12:45:06,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4439.21 MB 2025-02-15 12:45:06,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27237.81 MB 2025-02-15 12:45:06,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27237.81 MB 2025-02-15 12:45:06,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:06,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24988.11 MB 2025-02-15 12:45:06,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:45:06,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:45:06,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:45:06,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:06,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23893.19 MB 2025-02-15 12:45:06,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20337.13 MB 2025-02-15 12:45:06,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3556.07 MB 2025-02-15 12:45:06,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27237.81 MB 2025-02-15 12:45:06,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27237.81 MB 2025-02-15 12:45:06,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:06,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24988.11 MB 2025-02-15 12:45:07,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:45:07,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:45:07,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:45:07,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:07,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20668.02 MB 2025-02-15 12:45:07,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21027.28 MB 2025-02-15 12:45:07,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 359.26 MB 2025-02-15 12:45:07,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27237.81 MB 2025-02-15 12:45:07,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27432.85 MB 2025-02-15 12:45:07,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-15 12:45:07,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21362.39 MB 2025-02-15 12:45:07,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:45:07,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:45:07,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:07,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:07,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21220.31 MB 2025-02-15 12:45:07,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21421.76 MB 2025-02-15 12:45:07,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.45 MB 2025-02-15 12:45:07,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27432.85 MB 2025-02-15 12:45:07,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27432.85 MB 2025-02-15 12:45:07,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:07,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21451.18 MB 2025-02-15 12:45:07,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:45:07,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:45:07,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.00 seconds 2025-02-15 12:45:07,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:07,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22010.00 MB 2025-02-15 12:45:07,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21622.84 MB 2025-02-15 12:45:07,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -387.17 MB 2025-02-15 12:45:07,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39998.98 MB 2025-02-15 12:45:07,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27432.85 MB 2025-02-15 12:45:07,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12566.13 MB 2025-02-15 12:45:07,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21622.84 MB 2025-02-15 12:45:07,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:45:07,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:45:07,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:45:07,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:07,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21622.84 MB 2025-02-15 12:45:07,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21723.30 MB 2025-02-15 12:45:07,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:45:07,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27432.85 MB 2025-02-15 12:45:07,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27432.85 MB 2025-02-15 12:45:07,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:07,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22326.10 MB 2025-02-15 12:45:07,339 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:45:07,340 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:07,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:45:07,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:45:07,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:45:07,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:07,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17220.83 MB 2025-02-15 12:45:07,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21415.32 MB 2025-02-15 12:45:07,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:45:07,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27432.85 MB 2025-02-15 12:45:07,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31629.25 MB 2025-02-15 12:45:07,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 12:45:07,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25609.29 MB 2025-02-15 12:45:07,505 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:45:07,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,508 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:45:07,512 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:45:07,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,513 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:45:07,513 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:07,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,514 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,515 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:07,520 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:45:07,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,521 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,521 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:07,521 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:45:07,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,522 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:07,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,522 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:45:07,522 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:45:07,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,523 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:07,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,526 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,526 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,527 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:07,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:07,529 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:12,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:12,506 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:12,513 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:45:12,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:12,515 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:45:12,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:12,517 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:45:15,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:45:15,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:45:15,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-15 12:45:15,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:15,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17282.00 MB 2025-02-15 12:45:15,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17936.71 MB 2025-02-15 12:45:15,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-15 12:45:15,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31629.25 MB 2025-02-15 12:45:15,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24039.65 MB 2025-02-15 12:45:15,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7589.59 MB 2025-02-15 12:45:15,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26753.37 MB 2025-02-15 12:45:15,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:45:15,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:45:15,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:15,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:15,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17936.71 MB 2025-02-15 12:45:15,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18211.71 MB 2025-02-15 12:45:15,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 275.00 MB 2025-02-15 12:45:15,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24039.65 MB 2025-02-15 12:45:15,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24039.65 MB 2025-02-15 12:45:15,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:15,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20450.95 MB 2025-02-15 12:45:16,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:45:16,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:45:16,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 12:45:16,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18211.71 MB 2025-02-15 12:45:16,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18449.26 MB 2025-02-15 12:45:16,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.55 MB 2025-02-15 12:45:16,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24039.65 MB 2025-02-15 12:45:16,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23666.36 MB 2025-02-15 12:45:16,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -373.29 MB 2025-02-15 12:45:16,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22381.95 MB 2025-02-15 12:45:16,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:45:16,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:45:16,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:16,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18449.26 MB 2025-02-15 12:45:16,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19294.62 MB 2025-02-15 12:45:16,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.36 MB 2025-02-15 12:45:16,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23666.36 MB 2025-02-15 12:45:16,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23668.46 MB 2025-02-15 12:45:16,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 12:45:16,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19928.93 MB 2025-02-15 12:45:16,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:45:16,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:45:16,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:45:16,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19294.62 MB 2025-02-15 12:45:16,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20297.89 MB 2025-02-15 12:45:16,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.27 MB 2025-02-15 12:45:16,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23668.46 MB 2025-02-15 12:45:16,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23880.27 MB 2025-02-15 12:45:16,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 211.81 MB 2025-02-15 12:45:16,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22779.57 MB 2025-02-15 12:45:16,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:45:16,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:45:16,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:45:16,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18449.26 MB 2025-02-15 12:45:16,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20297.89 MB 2025-02-15 12:45:16,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1848.63 MB 2025-02-15 12:45:16,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23666.36 MB 2025-02-15 12:45:16,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23880.27 MB 2025-02-15 12:45:16,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 12:45:16,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22779.57 MB 2025-02-15 12:45:16,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:45:16,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:45:16,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:45:16,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20614.62 MB 2025-02-15 12:45:16,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20958.51 MB 2025-02-15 12:45:16,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 343.89 MB 2025-02-15 12:45:16,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23880.27 MB 2025-02-15 12:45:16,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24066.92 MB 2025-02-15 12:45:16,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-15 12:45:16,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21280.57 MB 2025-02-15 12:45:16,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:45:16,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:45:16,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:16,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21143.29 MB 2025-02-15 12:45:16,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21345.00 MB 2025-02-15 12:45:16,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.71 MB 2025-02-15 12:45:16,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24066.92 MB 2025-02-15 12:45:16,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24066.92 MB 2025-02-15 12:45:16,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:16,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21370.17 MB 2025-02-15 12:45:16,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:45:16,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:45:16,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.94 seconds 2025-02-15 12:45:16,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16637.45 MB 2025-02-15 12:45:16,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21545.85 MB 2025-02-15 12:45:16,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4908.40 MB 2025-02-15 12:45:16,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31629.25 MB 2025-02-15 12:45:16,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24066.92 MB 2025-02-15 12:45:16,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7562.33 MB 2025-02-15 12:45:16,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21545.85 MB 2025-02-15 12:45:16,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:45:16,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:45:16,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:45:16,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21545.85 MB 2025-02-15 12:45:16,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21646.09 MB 2025-02-15 12:45:16,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.25 MB 2025-02-15 12:45:16,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24066.92 MB 2025-02-15 12:45:16,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24066.92 MB 2025-02-15 12:45:16,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:16,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22247.57 MB 2025-02-15 12:45:16,741 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 12:45:16,741 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:16,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:45:16,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:45:16,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:45:16,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:16,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17314.10 MB 2025-02-15 12:45:16,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21500.01 MB 2025-02-15 12:45:16,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-15 12:45:16,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24066.92 MB 2025-02-15 12:45:16,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32438.75 MB 2025-02-15 12:45:16,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 12:45:16,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25684.75 MB 2025-02-15 12:45:16,906 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 12:45:16,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,907 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,908 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:45:16,912 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:45:16,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,913 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:45:16,914 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:16,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,914 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,915 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:16,921 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:45:16,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,921 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,922 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:16,922 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:45:16,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,922 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:16,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,923 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:45:16,923 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:45:16,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,923 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:16,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,926 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,927 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,927 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:16,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:16,929 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:23,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:23,485 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:23,489 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:45:23,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:23,491 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 102, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:45:23,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:23,491 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 102, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:45:25,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:45:25,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:45:25,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-15 12:45:25,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16825.42 MB 2025-02-15 12:45:25,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17186.39 MB 2025-02-15 12:45:25,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 360.97 MB 2025-02-15 12:45:25,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32438.75 MB 2025-02-15 12:45:25,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18987.61 MB 2025-02-15 12:45:25,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13451.13 MB 2025-02-15 12:45:25,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26070.30 MB 2025-02-15 12:45:25,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:45:25,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:45:25,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:45:25,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17186.39 MB 2025-02-15 12:45:25,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17361.28 MB 2025-02-15 12:45:25,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 174.89 MB 2025-02-15 12:45:25,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18987.61 MB 2025-02-15 12:45:25,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18987.61 MB 2025-02-15 12:45:25,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:25,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17902.81 MB 2025-02-15 12:45:25,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:45:25,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:45:25,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.49 seconds 2025-02-15 12:45:25,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17361.28 MB 2025-02-15 12:45:25,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17496.65 MB 2025-02-15 12:45:25,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.36 MB 2025-02-15 12:45:25,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18987.61 MB 2025-02-15 12:45:25,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19226.69 MB 2025-02-15 12:45:25,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 239.08 MB 2025-02-15 12:45:25,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21448.08 MB 2025-02-15 12:45:25,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:45:25,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:45:25,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:45:25,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17496.58 MB 2025-02-15 12:45:25,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17978.82 MB 2025-02-15 12:45:25,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 482.24 MB 2025-02-15 12:45:25,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19226.69 MB 2025-02-15 12:45:25,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19467.86 MB 2025-02-15 12:45:25,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 241.17 MB 2025-02-15 12:45:25,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18340.27 MB 2025-02-15 12:45:25,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:45:25,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:45:25,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:45:25,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17978.82 MB 2025-02-15 12:45:25,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18564.44 MB 2025-02-15 12:45:25,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 585.61 MB 2025-02-15 12:45:25,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19467.86 MB 2025-02-15 12:45:25,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20914.90 MB 2025-02-15 12:45:25,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1447.03 MB 2025-02-15 12:45:25,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19966.91 MB 2025-02-15 12:45:25,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:45:25,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:45:25,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:45:25,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17496.58 MB 2025-02-15 12:45:25,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18564.44 MB 2025-02-15 12:45:25,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1067.85 MB 2025-02-15 12:45:25,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19226.69 MB 2025-02-15 12:45:25,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20914.90 MB 2025-02-15 12:45:25,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1688.21 MB 2025-02-15 12:45:25,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19966.91 MB 2025-02-15 12:45:25,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:45:25,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:45:25,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:45:25,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18825.14 MB 2025-02-15 12:45:25,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19070.86 MB 2025-02-15 12:45:25,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.72 MB 2025-02-15 12:45:25,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20914.90 MB 2025-02-15 12:45:25,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-15 12:45:25,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 157.29 MB 2025-02-15 12:45:25,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19251.35 MB 2025-02-15 12:45:25,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:45:25,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:45:25,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:45:25,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19226.59 MB 2025-02-15 12:45:25,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19425.99 MB 2025-02-15 12:45:25,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.40 MB 2025-02-15 12:45:25,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-15 12:45:25,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-15 12:45:25,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:25,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19425.99 MB 2025-02-15 12:45:25,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:45:25,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:45:25,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.25 seconds 2025-02-15 12:45:25,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:25,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16470.05 MB 2025-02-15 12:45:25,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19626.30 MB 2025-02-15 12:45:25,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3156.25 MB 2025-02-15 12:45:25,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32438.75 MB 2025-02-15 12:45:25,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-15 12:45:25,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11366.56 MB 2025-02-15 12:45:25,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19626.30 MB 2025-02-15 12:45:26,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:45:26,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:45:26,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:45:26,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:26,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16840.99 MB 2025-02-15 12:45:26,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16941.62 MB 2025-02-15 12:45:26,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.63 MB 2025-02-15 12:45:26,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-15 12:45:26,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-15 12:45:26,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:26,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17542.13 MB 2025-02-15 12:45:26,026 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 12:45:26,026 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:45:26,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:45:26,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:45:26,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:45:26,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:26,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16941.62 MB 2025-02-15 12:45:26,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21121.24 MB 2025-02-15 12:45:26,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-15 12:45:26,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-15 12:45:26,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31522.29 MB 2025-02-15 12:45:26,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10450.11 MB 2025-02-15 12:45:26,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25299.31 MB 2025-02-15 12:45:26,189 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 12:45:26,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,191 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,192 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:45:26,196 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:45:26,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,197 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:45:26,197 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:45:26,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,198 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,199 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:26,204 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:45:26,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,205 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,205 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:26,205 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:45:26,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,206 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:26,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,206 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:45:26,206 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:45:26,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,207 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:26,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,210 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,210 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,211 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:26,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:26,213 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:36,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:36,584 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:36,589 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:45:36,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:36,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 151, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:45:36,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:36,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 151, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:45:38,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:45:38,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:45:38,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.34 seconds 2025-02-15 12:45:38,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:38,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22418.79 MB 2025-02-15 12:45:38,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22953.17 MB 2025-02-15 12:45:38,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 534.38 MB 2025-02-15 12:45:38,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31522.29 MB 2025-02-15 12:45:38,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27596.42 MB 2025-02-15 12:45:38,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3925.87 MB 2025-02-15 12:45:38,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31890.16 MB 2025-02-15 12:45:38,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:45:38,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:45:38,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:38,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:38,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22953.17 MB 2025-02-15 12:45:38,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23169.94 MB 2025-02-15 12:45:38,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.77 MB 2025-02-15 12:45:38,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27596.42 MB 2025-02-15 12:45:38,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27596.42 MB 2025-02-15 12:45:38,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:38,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24989.91 MB 2025-02-15 12:45:39,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:45:39,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:45:39,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-15 12:45:39,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23169.94 MB 2025-02-15 12:45:39,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23362.37 MB 2025-02-15 12:45:39,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 12:45:39,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27596.42 MB 2025-02-15 12:45:39,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27596.42 MB 2025-02-15 12:45:39,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:39,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27340.63 MB 2025-02-15 12:45:39,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:45:39,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:45:39,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:45:39,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23362.30 MB 2025-02-15 12:45:39,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24047.09 MB 2025-02-15 12:45:39,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 12:45:39,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27596.42 MB 2025-02-15 12:45:39,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27596.42 MB 2025-02-15 12:45:39,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:39,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24560.92 MB 2025-02-15 12:45:39,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:45:39,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:45:39,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:45:39,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24047.09 MB 2025-02-15 12:45:39,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24859.81 MB 2025-02-15 12:45:39,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 12:45:39,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27596.42 MB 2025-02-15 12:45:39,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28284.29 MB 2025-02-15 12:45:39,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 687.87 MB 2025-02-15 12:45:39,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.57 MB 2025-02-15 12:45:39,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:45:39,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:45:39,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:45:39,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23362.30 MB 2025-02-15 12:45:39,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24859.81 MB 2025-02-15 12:45:39,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 12:45:39,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27596.42 MB 2025-02-15 12:45:39,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28284.29 MB 2025-02-15 12:45:39,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 687.87 MB 2025-02-15 12:45:39,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.57 MB 2025-02-15 12:45:39,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:45:39,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:45:39,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:45:39,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25116.38 MB 2025-02-15 12:45:39,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25394.42 MB 2025-02-15 12:45:39,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-15 12:45:39,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28284.29 MB 2025-02-15 12:45:39,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28428.99 MB 2025-02-15 12:45:39,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-15 12:45:39,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25659.94 MB 2025-02-15 12:45:39,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:45:39,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:45:39,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:39,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25544.10 MB 2025-02-15 12:45:39,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25750.34 MB 2025-02-15 12:45:39,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.24 MB 2025-02-15 12:45:39,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28428.99 MB 2025-02-15 12:45:39,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28428.99 MB 2025-02-15 12:45:39,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:39,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25759.47 MB 2025-02-15 12:45:39,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:45:39,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:45:39,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-15 12:45:39,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:39,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.69 MB 2025-02-15 12:45:39,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25951.21 MB 2025-02-15 12:45:39,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4058.52 MB 2025-02-15 12:45:39,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31522.29 MB 2025-02-15 12:45:39,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28428.99 MB 2025-02-15 12:45:39,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3093.30 MB 2025-02-15 12:45:39,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25951.21 MB 2025-02-15 12:45:40,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:45:40,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:45:40,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:45:40,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:40,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25951.21 MB 2025-02-15 12:45:40,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26051.58 MB 2025-02-15 12:45:40,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 12:45:40,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28428.99 MB 2025-02-15 12:45:40,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28428.99 MB 2025-02-15 12:45:40,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:40,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26653.79 MB 2025-02-15 12:45:40,087 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 12:45:40,087 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:40,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:45:40,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:45:40,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:45:40,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:40,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22478.45 MB 2025-02-15 12:45:40,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26668.83 MB 2025-02-15 12:45:40,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 12:45:40,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28428.99 MB 2025-02-15 12:45:40,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-15 12:45:40,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-15 12:45:40,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30858.94 MB 2025-02-15 12:45:40,254 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 12:45:40,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:45:40,261 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:45:40,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,262 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:45:40,262 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:45:40,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,262 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,263 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:40,269 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:45:40,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,269 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,270 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:40,270 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:45:40,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,270 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:40,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,271 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:45:40,271 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:45:40,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,271 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:40,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,274 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,275 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,276 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:40,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:40,278 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:47,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:47,306 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:47,310 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:45:47,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:47,311 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:45:47,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:47,312 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:45:50,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:45:50,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:45:50,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.31 seconds 2025-02-15 12:45:50,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:50,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22979.11 MB 2025-02-15 12:45:50,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23736.44 MB 2025-02-15 12:45:50,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 757.33 MB 2025-02-15 12:45:50,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-15 12:45:50,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:50,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7109.35 MB 2025-02-15 12:45:50,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32676.97 MB 2025-02-15 12:45:50,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:45:50,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:45:50,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:50,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:50,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23736.44 MB 2025-02-15 12:45:50,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24068.19 MB 2025-02-15 12:45:50,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 331.75 MB 2025-02-15 12:45:50,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:50,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:50,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:50,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26672.06 MB 2025-02-15 12:45:51,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:45:51,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:45:51,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-15 12:45:51,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24068.19 MB 2025-02-15 12:45:51,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24345.55 MB 2025-02-15 12:45:51,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-15 12:45:51,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:51,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:51,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:51,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28322.77 MB 2025-02-15 12:45:51,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:45:51,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:45:51,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:51,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24345.55 MB 2025-02-15 12:45:51,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25332.59 MB 2025-02-15 12:45:51,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-15 12:45:51,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:51,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:51,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:51,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26073.20 MB 2025-02-15 12:45:51,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:45:51,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:45:51,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:45:51,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25332.59 MB 2025-02-15 12:45:51,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21375.54 MB 2025-02-15 12:45:51,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3957.06 MB 2025-02-15 12:45:51,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:51,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:51,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:51,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25947.12 MB 2025-02-15 12:45:51,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:45:51,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:45:51,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:45:51,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24345.55 MB 2025-02-15 12:45:51,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21375.54 MB 2025-02-15 12:45:51,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2970.01 MB 2025-02-15 12:45:51,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:51,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31799.12 MB 2025-02-15 12:45:51,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:51,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25947.12 MB 2025-02-15 12:45:51,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:45:51,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:45:51,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:45:51,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21745.36 MB 2025-02-15 12:45:51,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22146.12 MB 2025-02-15 12:45:51,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.76 MB 2025-02-15 12:45:51,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31799.12 MB 2025-02-15 12:45:51,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-15 12:45:51,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 209.72 MB 2025-02-15 12:45:51,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22517.93 MB 2025-02-15 12:45:51,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:45:51,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:45:51,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:45:51,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22361.86 MB 2025-02-15 12:45:51,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22569.15 MB 2025-02-15 12:45:51,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.30 MB 2025-02-15 12:45:51,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32008.83 MB 2025-02-15 12:45:51,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-15 12:45:51,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:51,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22611.59 MB 2025-02-15 12:45:51,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:45:51,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:45:51,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.54 seconds 2025-02-15 12:45:51,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:51,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22233.51 MB 2025-02-15 12:45:51,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22770.23 MB 2025-02-15 12:45:51,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 536.71 MB 2025-02-15 12:45:51,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-15 12:45:51,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-15 12:45:51,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6899.63 MB 2025-02-15 12:45:51,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22770.23 MB 2025-02-15 12:45:52,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:45:52,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:45:52,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:45:52,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:52,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17760.45 MB 2025-02-15 12:45:52,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17860.91 MB 2025-02-15 12:45:52,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:45:52,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32008.83 MB 2025-02-15 12:45:52,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-15 12:45:52,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:45:52,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18463.71 MB 2025-02-15 12:45:52,135 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:45:52,135 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for the video is 2.'] 2025-02-15 12:45:52,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:45:52,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:45:52,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:45:52,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:45:52,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17860.91 MB 2025-02-15 12:45:52,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22055.40 MB 2025-02-15 12:45:52,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:45:52,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32008.83 MB 2025-02-15 12:45:52,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40399.54 MB 2025-02-15 12:45:52,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:45:52,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26249.70 MB 2025-02-15 12:45:52,300 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:45:52,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,302 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:45:52,307 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:45:52,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,308 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:45:52,308 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for the video is 2.'] 2025-02-15 12:45:52,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,309 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,310 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:52,315 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:45:52,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,316 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,316 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:52,316 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:45:52,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,317 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:52,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,317 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:45:52,317 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:45:52,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,318 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:45:52,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,321 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,322 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,323 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:45:52,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:45:52,325 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:00,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:00,994 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:00,999 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:46:01,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:01,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:46:01,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:01,001 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:46:03,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:46:03,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:46:03,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-15 12:46:03,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:03,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17582.16 MB 2025-02-15 12:46:03,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18141.31 MB 2025-02-15 12:46:03,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-15 12:46:03,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40399.54 MB 2025-02-15 12:46:03,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:03,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13247.71 MB 2025-02-15 12:46:03,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27053.53 MB 2025-02-15 12:46:03,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:46:03,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:46:03,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:03,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:03,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18141.31 MB 2025-02-15 12:46:03,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18412.22 MB 2025-02-15 12:46:03,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.91 MB 2025-02-15 12:46:03,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:03,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:03,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:03,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20360.65 MB 2025-02-15 12:46:04,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:46:04,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:46:04,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.76 seconds 2025-02-15 12:46:04,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18412.22 MB 2025-02-15 12:46:04,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18621.90 MB 2025-02-15 12:46:04,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.68 MB 2025-02-15 12:46:04,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:04,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:04,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22581.87 MB 2025-02-15 12:46:04,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:46:04,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:46:04,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:46:04,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18621.84 MB 2025-02-15 12:46:04,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19368.02 MB 2025-02-15 12:46:04,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.18 MB 2025-02-15 12:46:04,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:04,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:04,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19927.91 MB 2025-02-15 12:46:04,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:46:04,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:46:04,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:46:04,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19368.02 MB 2025-02-15 12:46:04,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20253.60 MB 2025-02-15 12:46:04,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 885.57 MB 2025-02-15 12:46:04,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:04,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:04,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22443.55 MB 2025-02-15 12:46:04,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:46:04,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:46:04,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:46:04,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18621.84 MB 2025-02-15 12:46:04,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20253.60 MB 2025-02-15 12:46:04,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1631.76 MB 2025-02-15 12:46:04,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:04,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27151.83 MB 2025-02-15 12:46:04,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22443.55 MB 2025-02-15 12:46:04,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:46:04,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:46:04,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:46:04,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20533.17 MB 2025-02-15 12:46:04,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20836.14 MB 2025-02-15 12:46:04,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.97 MB 2025-02-15 12:46:04,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27151.83 MB 2025-02-15 12:46:04,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27315.40 MB 2025-02-15 12:46:04,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 12:46:04,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21122.84 MB 2025-02-15 12:46:04,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:46:04,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:46:04,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:04,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20999.24 MB 2025-02-15 12:46:04,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21200.60 MB 2025-02-15 12:46:04,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.36 MB 2025-02-15 12:46:04,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27315.40 MB 2025-02-15 12:46:04,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27315.40 MB 2025-02-15 12:46:04,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21213.25 MB 2025-02-15 12:46:04,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:46:04,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:46:04,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-15 12:46:04,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17031.67 MB 2025-02-15 12:46:04,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21401.55 MB 2025-02-15 12:46:04,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4369.87 MB 2025-02-15 12:46:04,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40399.54 MB 2025-02-15 12:46:04,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27315.40 MB 2025-02-15 12:46:04,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13084.13 MB 2025-02-15 12:46:04,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21401.55 MB 2025-02-15 12:46:04,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:46:04,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:46:04,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:46:04,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21401.55 MB 2025-02-15 12:46:04,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21501.95 MB 2025-02-15 12:46:04,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.41 MB 2025-02-15 12:46:04,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27315.40 MB 2025-02-15 12:46:04,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27315.40 MB 2025-02-15 12:46:04,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:04,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22104.38 MB 2025-02-15 12:46:04,673 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 12:46:04,673 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:46:04,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:46:04,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:46:04,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:46:04,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:04,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17652.02 MB 2025-02-15 12:46:04,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21843.94 MB 2025-02-15 12:46:04,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4191.92 MB 2025-02-15 12:46:04,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27315.40 MB 2025-02-15 12:46:04,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31507.61 MB 2025-02-15 12:46:04,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 12:46:04,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26036.14 MB 2025-02-15 12:46:04,840 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 12:46:04,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,841 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:46:04,847 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:46:04,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,848 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:46:04,848 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:46:04,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,848 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,849 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:04,855 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:46:04,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,855 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,856 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:04,856 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:46:04,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,856 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:04,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,857 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:46:04,857 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:46:04,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,857 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:04,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,861 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,863 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,864 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:04,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:04,867 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:13,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:13,747 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:13,752 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:46:13,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:13,753 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:46:13,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:13,754 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:46:16,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:46:16,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:46:16,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-15 12:46:16,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:16,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23209.42 MB 2025-02-15 12:46:16,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23779.19 MB 2025-02-15 12:46:16,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-15 12:46:16,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31507.61 MB 2025-02-15 12:46:16,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:16,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -163.58 MB 2025-02-15 12:46:16,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32680.79 MB 2025-02-15 12:46:16,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:46:16,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:46:16,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:16,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:16,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23779.19 MB 2025-02-15 12:46:16,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24020.13 MB 2025-02-15 12:46:16,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.94 MB 2025-02-15 12:46:16,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:16,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:16,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:16,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25970.44 MB 2025-02-15 12:46:17,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:46:17,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:46:17,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.75 seconds 2025-02-15 12:46:17,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24020.13 MB 2025-02-15 12:46:17,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24227.16 MB 2025-02-15 12:46:17,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.03 MB 2025-02-15 12:46:17,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:17,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:17,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28189.78 MB 2025-02-15 12:46:17,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:46:17,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:46:17,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:46:17,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24227.09 MB 2025-02-15 12:46:17,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24963.83 MB 2025-02-15 12:46:17,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.74 MB 2025-02-15 12:46:17,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:17,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:17,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25516.64 MB 2025-02-15 12:46:17,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:46:17,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:46:17,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:46:17,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24963.83 MB 2025-02-15 12:46:17,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20353.57 MB 2025-02-15 12:46:17,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4610.26 MB 2025-02-15 12:46:17,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:17,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:17,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25309.54 MB 2025-02-15 12:46:17,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:46:17,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:46:17,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:46:17,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24227.09 MB 2025-02-15 12:46:17,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20353.57 MB 2025-02-15 12:46:17,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3873.52 MB 2025-02-15 12:46:17,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:17,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-15 12:46:17,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25309.54 MB 2025-02-15 12:46:17,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:46:17,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:46:17,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:46:17,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20629.61 MB 2025-02-15 12:46:17,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20928.74 MB 2025-02-15 12:46:17,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.13 MB 2025-02-15 12:46:17,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-15 12:46:17,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31505.51 MB 2025-02-15 12:46:17,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-15 12:46:17,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21213.82 MB 2025-02-15 12:46:17,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:46:17,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:46:17,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:17,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21089.77 MB 2025-02-15 12:46:17,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21291.24 MB 2025-02-15 12:46:17,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.47 MB 2025-02-15 12:46:17,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31505.51 MB 2025-02-15 12:46:17,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31505.51 MB 2025-02-15 12:46:17,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21300.60 MB 2025-02-15 12:46:17,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:46:17,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:46:17,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.43 seconds 2025-02-15 12:46:17,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22648.49 MB 2025-02-15 12:46:17,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21492.27 MB 2025-02-15 12:46:17,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1156.22 MB 2025-02-15 12:46:17,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31507.61 MB 2025-02-15 12:46:17,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31505.51 MB 2025-02-15 12:46:17,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2.10 MB 2025-02-15 12:46:17,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21492.27 MB 2025-02-15 12:46:17,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:46:17,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:46:17,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:46:17,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21492.27 MB 2025-02-15 12:46:17,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21592.71 MB 2025-02-15 12:46:17,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-15 12:46:17,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31505.51 MB 2025-02-15 12:46:17,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31505.51 MB 2025-02-15 12:46:17,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:17,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22195.36 MB 2025-02-15 12:46:17,461 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 12:46:17,462 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for the video is 2.'] 2025-02-15 12:46:17,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:46:17,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:46:17,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:46:17,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:17,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17778.97 MB 2025-02-15 12:46:17,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21972.42 MB 2025-02-15 12:46:17,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4193.46 MB 2025-02-15 12:46:17,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31505.51 MB 2025-02-15 12:46:17,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35699.82 MB 2025-02-15 12:46:17,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 12:46:17,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26165.37 MB 2025-02-15 12:46:17,627 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 12:46:17,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,628 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,629 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:46:17,634 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:46:17,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,635 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:46:17,635 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for the video is 2.'] 2025-02-15 12:46:17,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,635 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,636 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:17,642 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:46:17,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,642 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,643 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:17,643 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:46:17,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,643 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:17,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,644 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:46:17,644 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:46:17,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,644 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:17,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,648 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,649 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,650 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:17,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:17,653 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:26,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:26,464 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:26,469 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:46:26,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:26,470 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 194, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:46:26,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:26,471 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 194, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:46:29,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:46:29,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:46:29,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-15 12:46:29,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:29,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23567.72 MB 2025-02-15 12:46:29,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24254.28 MB 2025-02-15 12:46:29,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 686.56 MB 2025-02-15 12:46:29,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35699.82 MB 2025-02-15 12:46:29,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28789.70 MB 2025-02-15 12:46:29,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6910.12 MB 2025-02-15 12:46:29,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33265.59 MB 2025-02-15 12:46:29,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:46:29,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:46:29,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:29,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:29,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24254.28 MB 2025-02-15 12:46:29,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24579.82 MB 2025-02-15 12:46:29,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.55 MB 2025-02-15 12:46:29,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28789.70 MB 2025-02-15 12:46:29,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28789.70 MB 2025-02-15 12:46:29,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:29,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26965.16 MB 2025-02-15 12:46:30,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:46:30,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:46:30,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 12:46:30,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24579.82 MB 2025-02-15 12:46:30,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24835.96 MB 2025-02-15 12:46:30,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 12:46:30,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28789.70 MB 2025-02-15 12:46:30,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28789.70 MB 2025-02-15 12:46:30,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:30,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28835.45 MB 2025-02-15 12:46:30,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:46:30,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:46:30,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:30,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24835.96 MB 2025-02-15 12:46:30,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25747.43 MB 2025-02-15 12:46:30,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 12:46:30,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28789.70 MB 2025-02-15 12:46:30,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28789.70 MB 2025-02-15 12:46:30,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:30,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26431.35 MB 2025-02-15 12:46:30,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:46:30,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:46:30,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:46:30,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25747.43 MB 2025-02-15 12:46:30,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26829.16 MB 2025-02-15 12:46:30,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 12:46:30,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28789.70 MB 2025-02-15 12:46:30,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31075.60 MB 2025-02-15 12:46:30,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 12:46:30,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29504.25 MB 2025-02-15 12:46:30,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:46:30,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:46:30,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:46:30,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24835.96 MB 2025-02-15 12:46:30,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26829.16 MB 2025-02-15 12:46:30,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 12:46:30,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28789.70 MB 2025-02-15 12:46:30,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31075.60 MB 2025-02-15 12:46:30,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 12:46:30,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29504.25 MB 2025-02-15 12:46:30,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:46:30,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:46:30,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:46:30,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27170.67 MB 2025-02-15 12:46:30,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27540.75 MB 2025-02-15 12:46:30,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 12:46:30,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31075.60 MB 2025-02-15 12:46:30,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:46:30,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-15 12:46:30,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27885.94 MB 2025-02-15 12:46:30,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:46:30,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:46:30,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:30,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27739.98 MB 2025-02-15 12:46:30,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27946.26 MB 2025-02-15 12:46:30,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.28 MB 2025-02-15 12:46:30,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:46:30,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:46:30,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:30,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27986.46 MB 2025-02-15 12:46:30,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:46:30,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:46:30,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.22 seconds 2025-02-15 12:46:30,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22891.81 MB 2025-02-15 12:46:30,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28147.33 MB 2025-02-15 12:46:30,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5255.52 MB 2025-02-15 12:46:30,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35699.82 MB 2025-02-15 12:46:30,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:46:30,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4422.89 MB 2025-02-15 12:46:30,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28147.33 MB 2025-02-15 12:46:30,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:46:30,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:46:30,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:46:30,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28147.33 MB 2025-02-15 12:46:30,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28247.80 MB 2025-02-15 12:46:30,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:46:30,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:46:30,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:46:30,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:30,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28850.60 MB 2025-02-15 12:46:30,973 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:46:30,973 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 12:46:30,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:46:30,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:46:30,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:46:30,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:30,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23605.19 MB 2025-02-15 12:46:30,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27799.68 MB 2025-02-15 12:46:30,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:46:30,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:46:30,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41766.88 MB 2025-02-15 12:46:30,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:46:30,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31993.98 MB 2025-02-15 12:46:31,138 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:46:31,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,139 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,140 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:46:31,144 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:46:31,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,145 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:46:31,146 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 12:46:31,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,146 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,147 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:31,153 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:46:31,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,153 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,154 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:31,154 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:46:31,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,154 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:31,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,155 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:46:31,155 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:46:31,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,155 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:31,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,158 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,159 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,160 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:31,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:31,162 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:40,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:40,490 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:40,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:46:40,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:40,496 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:46:40,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:40,497 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:46:43,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:46:43,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:46:43,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-15 12:46:43,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:43,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23626.12 MB 2025-02-15 12:46:43,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24280.83 MB 2025-02-15 12:46:43,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-15 12:46:43,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41766.88 MB 2025-02-15 12:46:43,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36458.99 MB 2025-02-15 12:46:43,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5307.89 MB 2025-02-15 12:46:43,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33097.49 MB 2025-02-15 12:46:43,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:46:43,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:46:43,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:43,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:43,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24280.83 MB 2025-02-15 12:46:43,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24457.57 MB 2025-02-15 12:46:43,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 176.74 MB 2025-02-15 12:46:43,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36458.99 MB 2025-02-15 12:46:43,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36458.99 MB 2025-02-15 12:46:43,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:43,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26598.49 MB 2025-02-15 12:46:44,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:46:44,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:46:44,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-15 12:46:44,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24457.57 MB 2025-02-15 12:46:44,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24676.54 MB 2025-02-15 12:46:44,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-15 12:46:44,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36458.99 MB 2025-02-15 12:46:44,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-15 12:46:44,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -411.04 MB 2025-02-15 12:46:44,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28627.22 MB 2025-02-15 12:46:44,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:46:44,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:46:44,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:46:44,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24676.48 MB 2025-02-15 12:46:44,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25455.72 MB 2025-02-15 12:46:44,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 779.24 MB 2025-02-15 12:46:44,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-15 12:46:44,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-15 12:46:44,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26040.41 MB 2025-02-15 12:46:44,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:46:44,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:46:44,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:46:44,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25455.72 MB 2025-02-15 12:46:44,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20889.14 MB 2025-02-15 12:46:44,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4566.58 MB 2025-02-15 12:46:44,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-15 12:46:44,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-15 12:46:44,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25801.83 MB 2025-02-15 12:46:44,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:46:44,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:46:44,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:46:44,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24676.48 MB 2025-02-15 12:46:44,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20889.14 MB 2025-02-15 12:46:44,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3787.33 MB 2025-02-15 12:46:44,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-15 12:46:44,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-15 12:46:44,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25801.83 MB 2025-02-15 12:46:44,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:46:44,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:46:44,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:46:44,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21181.11 MB 2025-02-15 12:46:44,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21497.49 MB 2025-02-15 12:46:44,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 316.39 MB 2025-02-15 12:46:44,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-15 12:46:44,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36217.82 MB 2025-02-15 12:46:44,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-15 12:46:44,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21796.47 MB 2025-02-15 12:46:44,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:46:44,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:46:44,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:44,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21667.82 MB 2025-02-15 12:46:44,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21873.38 MB 2025-02-15 12:46:44,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.56 MB 2025-02-15 12:46:44,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36217.82 MB 2025-02-15 12:46:44,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36217.82 MB 2025-02-15 12:46:44,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21877.75 MB 2025-02-15 12:46:44,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:46:44,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:46:44,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.83 seconds 2025-02-15 12:46:44,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22981.57 MB 2025-02-15 12:46:44,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22074.40 MB 2025-02-15 12:46:44,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -907.17 MB 2025-02-15 12:46:44,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41766.88 MB 2025-02-15 12:46:44,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36217.82 MB 2025-02-15 12:46:44,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5549.06 MB 2025-02-15 12:46:44,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22074.40 MB 2025-02-15 12:46:44,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:46:44,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:46:44,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:46:44,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22074.40 MB 2025-02-15 12:46:44,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22174.84 MB 2025-02-15 12:46:44,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-15 12:46:44,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36217.82 MB 2025-02-15 12:46:44,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36217.82 MB 2025-02-15 12:46:44,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22777.50 MB 2025-02-15 12:46:44,613 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 12:46:44,614 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:46:44,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:46:44,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:46:44,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:46:44,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:44,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18129.18 MB 2025-02-15 12:46:44,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22323.67 MB 2025-02-15 12:46:44,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:46:44,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36217.82 MB 2025-02-15 12:46:44,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36217.82 MB 2025-02-15 12:46:44,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:44,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26516.61 MB 2025-02-15 12:46:44,777 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 12:46:44,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:46:44,784 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:46:44,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,785 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:46:44,785 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:46:44,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,786 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,786 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:44,792 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:46:44,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,793 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,793 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:44,793 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:46:44,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,794 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:44,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,794 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:46:44,794 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:46:44,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,795 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:44,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,798 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,799 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,800 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:44,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:44,803 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:53,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:53,942 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:53,947 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:46:53,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:53,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 208, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:46:53,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:53,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 208, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:46:57,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:46:57,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:46:57,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.23 seconds 2025-02-15 12:46:57,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:57,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18417.09 MB 2025-02-15 12:46:57,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19153.19 MB 2025-02-15 12:46:57,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.10 MB 2025-02-15 12:46:57,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36217.82 MB 2025-02-15 12:46:57,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23458.74 MB 2025-02-15 12:46:57,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12759.07 MB 2025-02-15 12:46:57,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28114.95 MB 2025-02-15 12:46:57,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:46:57,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:46:57,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:57,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:57,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19153.19 MB 2025-02-15 12:46:57,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19509.76 MB 2025-02-15 12:46:57,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.57 MB 2025-02-15 12:46:57,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23458.74 MB 2025-02-15 12:46:57,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24194.84 MB 2025-02-15 12:46:57,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 736.10 MB 2025-02-15 12:46:57,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22074.76 MB 2025-02-15 12:46:58,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:46:58,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:46:58,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-15 12:46:58,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19509.76 MB 2025-02-15 12:46:58,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19785.80 MB 2025-02-15 12:46:58,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-15 12:46:58,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24194.84 MB 2025-02-15 12:46:58,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23737.66 MB 2025-02-15 12:46:58,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -457.18 MB 2025-02-15 12:46:58,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23765.38 MB 2025-02-15 12:46:58,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:46:58,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:46:58,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:58,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19785.80 MB 2025-02-15 12:46:58,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20768.12 MB 2025-02-15 12:46:58,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-15 12:46:58,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23737.66 MB 2025-02-15 12:46:58,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23737.66 MB 2025-02-15 12:46:58,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:58,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21505.18 MB 2025-02-15 12:46:58,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:46:58,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:46:58,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:46:58,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20768.12 MB 2025-02-15 12:46:58,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21933.91 MB 2025-02-15 12:46:58,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1165.80 MB 2025-02-15 12:46:58,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23737.66 MB 2025-02-15 12:46:58,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26191.33 MB 2025-02-15 12:46:58,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-15 12:46:58,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24816.91 MB 2025-02-15 12:46:58,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:46:58,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:46:58,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:46:58,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19785.80 MB 2025-02-15 12:46:58,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21933.91 MB 2025-02-15 12:46:58,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.11 MB 2025-02-15 12:46:58,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23737.66 MB 2025-02-15 12:46:58,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26191.33 MB 2025-02-15 12:46:58,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-15 12:46:58,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24816.91 MB 2025-02-15 12:46:58,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:46:58,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:46:58,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:46:58,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22301.96 MB 2025-02-15 12:46:58,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22700.80 MB 2025-02-15 12:46:58,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-15 12:46:58,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26191.33 MB 2025-02-15 12:46:58,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26407.34 MB 2025-02-15 12:46:58,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-15 12:46:58,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23069.24 MB 2025-02-15 12:46:58,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:46:58,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:46:58,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:46:58,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22915.51 MB 2025-02-15 12:46:58,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23122.21 MB 2025-02-15 12:46:58,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.69 MB 2025-02-15 12:46:58,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26407.34 MB 2025-02-15 12:46:58,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26407.34 MB 2025-02-15 12:46:58,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:58,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23171.89 MB 2025-02-15 12:46:58,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:46:58,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:46:58,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.46 seconds 2025-02-15 12:46:58,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17692.40 MB 2025-02-15 12:46:58,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23322.79 MB 2025-02-15 12:46:58,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5630.39 MB 2025-02-15 12:46:58,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36217.82 MB 2025-02-15 12:46:58,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26407.34 MB 2025-02-15 12:46:58,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9810.48 MB 2025-02-15 12:46:58,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23322.79 MB 2025-02-15 12:46:58,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:46:58,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:46:58,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:46:58,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18344.89 MB 2025-02-15 12:46:58,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18445.11 MB 2025-02-15 12:46:58,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-15 12:46:58,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26407.34 MB 2025-02-15 12:46:58,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26407.34 MB 2025-02-15 12:46:58,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:46:58,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19046.70 MB 2025-02-15 12:46:58,690 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 12:46:58,690 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:46:58,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:46:58,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:46:58,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:46:58,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:46:58,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18445.11 MB 2025-02-15 12:46:58,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22629.33 MB 2025-02-15 12:46:58,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-15 12:46:58,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26407.34 MB 2025-02-15 12:46:58,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36870.03 MB 2025-02-15 12:46:58,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-15 12:46:58,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26813.15 MB 2025-02-15 12:46:58,855 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 12:46:58,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,856 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,857 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:46:58,862 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:46:58,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,863 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:46:58,863 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:46:58,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,864 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,864 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:58,870 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:46:58,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,870 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,871 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:58,871 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:46:58,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,871 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:58,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,872 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:46:58,872 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:46:58,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,872 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:46:58,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,875 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,876 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,878 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:46:58,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:46:58,881 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:04,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:04,616 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:04,621 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:47:04,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:04,622 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:47:04,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:04,623 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:47:07,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:47:07,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:47:07,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.74 seconds 2025-02-15 12:47:07,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:07,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18322.26 MB 2025-02-15 12:47:07,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18948.66 MB 2025-02-15 12:47:07,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-15 12:47:07,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36870.03 MB 2025-02-15 12:47:07,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:07,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9676.26 MB 2025-02-15 12:47:07,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27793.63 MB 2025-02-15 12:47:07,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:47:07,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:47:07,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:07,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:07,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18948.66 MB 2025-02-15 12:47:07,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19252.14 MB 2025-02-15 12:47:07,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 303.49 MB 2025-02-15 12:47:07,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:07,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:07,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:07,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21434.87 MB 2025-02-15 12:47:08,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:47:08,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:47:08,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-15 12:47:08,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19252.14 MB 2025-02-15 12:47:08,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19487.04 MB 2025-02-15 12:47:08,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 12:47:08,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:08,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:08,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23421.79 MB 2025-02-15 12:47:08,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:47:08,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:47:08,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:08,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19486.97 MB 2025-02-15 12:47:08,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20322.89 MB 2025-02-15 12:47:08,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 12:47:08,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:08,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:08,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20950.11 MB 2025-02-15 12:47:08,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:47:08,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:47:08,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:47:08,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20322.89 MB 2025-02-15 12:47:08,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21314.95 MB 2025-02-15 12:47:08,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-15 12:47:08,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:08,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:08,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23768.26 MB 2025-02-15 12:47:08,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:47:08,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:47:08,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:47:08,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19486.97 MB 2025-02-15 12:47:08,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21314.95 MB 2025-02-15 12:47:08,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-15 12:47:08,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:08,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27193.77 MB 2025-02-15 12:47:08,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23768.26 MB 2025-02-15 12:47:08,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:47:08,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:47:08,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:47:08,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21628.14 MB 2025-02-15 12:47:08,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21967.54 MB 2025-02-15 12:47:08,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-15 12:47:08,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27193.77 MB 2025-02-15 12:47:08,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:47:08,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 12:47:08,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22286.27 MB 2025-02-15 12:47:08,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:47:08,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:47:08,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:08,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22150.25 MB 2025-02-15 12:47:08,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22357.17 MB 2025-02-15 12:47:08,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.92 MB 2025-02-15 12:47:08,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:47:08,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:47:08,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22385.93 MB 2025-02-15 12:47:08,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:47:08,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:47:08,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.79 seconds 2025-02-15 12:47:08,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17705.58 MB 2025-02-15 12:47:08,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22558.24 MB 2025-02-15 12:47:08,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4852.66 MB 2025-02-15 12:47:08,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36870.03 MB 2025-02-15 12:47:08,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:47:08,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9493.81 MB 2025-02-15 12:47:08,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22558.24 MB 2025-02-15 12:47:08,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:47:08,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:47:08,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:47:08,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22558.24 MB 2025-02-15 12:47:08,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22658.71 MB 2025-02-15 12:47:08,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:47:08,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:47:08,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:47:08,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:08,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23261.51 MB 2025-02-15 12:47:08,702 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:47:08,702 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:47:08,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:47:08,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:47:08,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:47:08,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:08,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18376.48 MB 2025-02-15 12:47:08,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22570.97 MB 2025-02-15 12:47:08,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:47:08,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:47:08,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35766.93 MB 2025-02-15 12:47:08,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:47:08,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26765.27 MB 2025-02-15 12:47:08,871 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:47:08,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,873 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,873 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:47:08,878 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:47:08,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,879 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:47:08,879 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:47:08,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,880 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,881 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:08,886 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:47:08,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,887 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,887 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:08,887 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:47:08,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,888 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:08,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,888 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:47:08,888 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:47:08,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,889 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:08,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,894 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,895 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,896 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:08,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:08,900 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:24,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:24,579 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:24,584 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:47:24,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:24,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:47:24,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:24,586 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:47:26,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:47:26,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:47:26,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.13 seconds 2025-02-15 12:47:26,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:26,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18158.43 MB 2025-02-15 12:47:26,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18639.73 MB 2025-02-15 12:47:26,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-15 12:47:26,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35766.93 MB 2025-02-15 12:47:26,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:26,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4378.85 MB 2025-02-15 12:47:26,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27629.80 MB 2025-02-15 12:47:26,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:47:26,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:47:26,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:26,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:26,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18639.73 MB 2025-02-15 12:47:26,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18830.77 MB 2025-02-15 12:47:26,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-15 12:47:26,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:26,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:26,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:26,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20465.78 MB 2025-02-15 12:47:27,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:47:27,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:47:27,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-15 12:47:27,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18830.77 MB 2025-02-15 12:47:27,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19003.30 MB 2025-02-15 12:47:27,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-15 12:47:27,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:27,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:27,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:27,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23000.42 MB 2025-02-15 12:47:27,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:47:27,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:47:27,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:47:27,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19003.23 MB 2025-02-15 12:47:27,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19617.18 MB 2025-02-15 12:47:27,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-15 12:47:27,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:27,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:27,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:27,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20077.85 MB 2025-02-15 12:47:27,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:47:27,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:47:27,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:47:27,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19617.18 MB 2025-02-15 12:47:27,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20345.83 MB 2025-02-15 12:47:27,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-15 12:47:27,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:27,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:27,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:27,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22147.68 MB 2025-02-15 12:47:27,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:47:27,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:47:27,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:47:27,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19003.23 MB 2025-02-15 12:47:27,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20345.83 MB 2025-02-15 12:47:27,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-15 12:47:27,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:27,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-15 12:47:27,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:27,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22147.68 MB 2025-02-15 12:47:27,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:47:27,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:47:27,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:47:27,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20575.86 MB 2025-02-15 12:47:27,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20825.14 MB 2025-02-15 12:47:27,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-15 12:47:27,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31388.07 MB 2025-02-15 12:47:27,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31522.29 MB 2025-02-15 12:47:27,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-15 12:47:27,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21066.03 MB 2025-02-15 12:47:27,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:47:27,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:47:27,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:27,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20959.33 MB 2025-02-15 12:47:27,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21161.24 MB 2025-02-15 12:47:27,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.90 MB 2025-02-15 12:47:27,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31522.29 MB 2025-02-15 12:47:27,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31526.49 MB 2025-02-15 12:47:27,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:47:27,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21161.24 MB 2025-02-15 12:47:27,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:47:27,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:47:27,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.92 seconds 2025-02-15 12:47:27,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17684.59 MB 2025-02-15 12:47:27,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21361.97 MB 2025-02-15 12:47:27,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3677.37 MB 2025-02-15 12:47:27,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35766.93 MB 2025-02-15 12:47:27,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31526.49 MB 2025-02-15 12:47:27,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4240.44 MB 2025-02-15 12:47:27,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21361.97 MB 2025-02-15 12:47:27,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:47:27,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:47:27,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:47:27,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18130.09 MB 2025-02-15 12:47:27,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18230.38 MB 2025-02-15 12:47:27,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.29 MB 2025-02-15 12:47:27,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31526.49 MB 2025-02-15 12:47:27,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31526.49 MB 2025-02-15 12:47:27,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:27,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18832.15 MB 2025-02-15 12:47:27,790 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 12:47:27,790 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 12:47:27,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:47:27,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:47:27,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:47:27,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:27,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18230.38 MB 2025-02-15 12:47:27,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22417.68 MB 2025-02-15 12:47:27,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4187.30 MB 2025-02-15 12:47:27,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31526.49 MB 2025-02-15 12:47:27,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35714.50 MB 2025-02-15 12:47:27,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 12:47:27,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26604.47 MB 2025-02-15 12:47:27,961 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 12:47:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:47:27,968 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:47:27,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,969 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:47:27,969 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 12:47:27,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:27,976 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:47:27,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,977 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,977 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:27,977 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:47:27,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,978 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:27,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,978 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:47:27,978 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:47:27,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,979 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:27,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,983 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,984 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,986 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:27,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:27,989 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:33,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:33,223 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:33,228 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:47:33,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:33,230 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 359, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:47:33,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:33,230 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 359, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:47:38,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:47:38,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:47:38,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.56 seconds 2025-02-15 12:47:38,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:38,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19834.06 MB 2025-02-15 12:47:38,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21104.94 MB 2025-02-15 12:47:38,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1270.87 MB 2025-02-15 12:47:38,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-15 12:47:38,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23819.45 MB 2025-02-15 12:47:38,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11895.05 MB 2025-02-15 12:47:38,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29985.72 MB 2025-02-15 12:47:38,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:47:38,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:47:38,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:47:38,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:38,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21104.94 MB 2025-02-15 12:47:38,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21334.71 MB 2025-02-15 12:47:38,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.77 MB 2025-02-15 12:47:38,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23819.45 MB 2025-02-15 12:47:38,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27554.48 MB 2025-02-15 12:47:38,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3735.03 MB 2025-02-15 12:47:38,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25377.88 MB 2025-02-15 12:47:40,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:47:40,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:47:40,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.47 seconds 2025-02-15 12:47:40,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21334.71 MB 2025-02-15 12:47:40,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21738.15 MB 2025-02-15 12:47:40,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 403.44 MB 2025-02-15 12:47:40,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27554.48 MB 2025-02-15 12:47:40,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24146.61 MB 2025-02-15 12:47:40,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3407.87 MB 2025-02-15 12:47:40,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25674.23 MB 2025-02-15 12:47:40,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:47:40,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:47:40,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:40,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21738.15 MB 2025-02-15 12:47:40,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23174.69 MB 2025-02-15 12:47:40,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1436.55 MB 2025-02-15 12:47:40,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24146.61 MB 2025-02-15 12:47:40,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26298.29 MB 2025-02-15 12:47:40,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2151.68 MB 2025-02-15 12:47:40,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24251.94 MB 2025-02-15 12:47:40,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:47:40,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:47:40,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:47:40,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23174.69 MB 2025-02-15 12:47:40,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.52 MB 2025-02-15 12:47:40,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1703.83 MB 2025-02-15 12:47:40,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26298.29 MB 2025-02-15 12:47:40,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30960.25 MB 2025-02-15 12:47:40,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4661.97 MB 2025-02-15 12:47:40,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29092.16 MB 2025-02-15 12:47:40,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:47:40,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:47:40,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:47:40,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21738.15 MB 2025-02-15 12:47:40,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.52 MB 2025-02-15 12:47:40,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3140.38 MB 2025-02-15 12:47:40,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24146.61 MB 2025-02-15 12:47:40,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30960.25 MB 2025-02-15 12:47:40,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6813.65 MB 2025-02-15 12:47:40,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29092.16 MB 2025-02-15 12:47:40,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:47:40,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:47:40,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:47:40,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25416.44 MB 2025-02-15 12:47:40,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25999.36 MB 2025-02-15 12:47:40,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 582.92 MB 2025-02-15 12:47:40,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30960.25 MB 2025-02-15 12:47:40,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:47:40,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 316.67 MB 2025-02-15 12:47:40,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26537.28 MB 2025-02-15 12:47:40,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:47:40,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:47:40,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:47:40,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26313.16 MB 2025-02-15 12:47:40,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26519.77 MB 2025-02-15 12:47:40,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.61 MB 2025-02-15 12:47:40,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:47:40,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:47:40,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:40,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26625.09 MB 2025-02-15 12:47:40,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:47:40,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:47:40,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.39 seconds 2025-02-15 12:47:40,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18583.28 MB 2025-02-15 12:47:40,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26720.52 MB 2025-02-15 12:47:40,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8137.24 MB 2025-02-15 12:47:40,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-15 12:47:40,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:47:40,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4437.57 MB 2025-02-15 12:47:40,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26720.52 MB 2025-02-15 12:47:40,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:47:40,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:47:40,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:47:40,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26720.52 MB 2025-02-15 12:47:40,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26820.83 MB 2025-02-15 12:47:40,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.31 MB 2025-02-15 12:47:40,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:47:40,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31276.92 MB 2025-02-15 12:47:40,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:47:40,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27422.67 MB 2025-02-15 12:47:40,905 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 12:47:40,905 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:47:40,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:47:40,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:47:40,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:47:40,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:47:40,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26820.83 MB 2025-02-15 12:47:40,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23779.03 MB 2025-02-15 12:47:40,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3041.80 MB 2025-02-15 12:47:40,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31276.92 MB 2025-02-15 12:47:40,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41748.00 MB 2025-02-15 12:47:40,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 12:47:40,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27967.05 MB 2025-02-15 12:47:41,075 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 12:47:41,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,076 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:47:41,082 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:47:41,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,083 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:47:41,083 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:47:41,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,084 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,084 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:41,090 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:47:41,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,091 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,091 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:41,091 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:47:41,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,092 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:41,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,092 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:47:41,092 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:47:41,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,093 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:41,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,098 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,100 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,102 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:47:41,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:41,106 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:59,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:59,627 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:47:59,632 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:47:59,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:59,633 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:47:59,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:47:59,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:48:01,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:48:01,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:48:01,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.62 seconds 2025-02-15 12:48:01,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18178.78 MB 2025-02-15 12:48:01,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18546.83 MB 2025-02-15 12:48:01,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 368.05 MB 2025-02-15 12:48:01,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41748.00 MB 2025-02-15 12:48:01,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18836.62 MB 2025-02-15 12:48:01,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27423.66 MB 2025-02-15 12:48:01,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:48:01,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:48:01,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:01,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18546.83 MB 2025-02-15 12:48:01,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18725.15 MB 2025-02-15 12:48:01,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 178.32 MB 2025-02-15 12:48:01,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19277.29 MB 2025-02-15 12:48:01,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:48:01,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:48:01,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.50 seconds 2025-02-15 12:48:01,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18725.15 MB 2025-02-15 12:48:01,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18863.17 MB 2025-02-15 12:48:01,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 138.02 MB 2025-02-15 12:48:01,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22810.90 MB 2025-02-15 12:48:01,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:48:01,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:48:01,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:01,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18863.10 MB 2025-02-15 12:48:01,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19354.26 MB 2025-02-15 12:48:01,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 491.16 MB 2025-02-15 12:48:01,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19722.80 MB 2025-02-15 12:48:01,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:48:01,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:48:01,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:48:01,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19354.26 MB 2025-02-15 12:48:01,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19950.82 MB 2025-02-15 12:48:01,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 596.56 MB 2025-02-15 12:48:01,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21378.66 MB 2025-02-15 12:48:01,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:48:01,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:48:01,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:48:01,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18863.17 MB 2025-02-15 12:48:01,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19950.82 MB 2025-02-15 12:48:01,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1087.66 MB 2025-02-15 12:48:01,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22911.39 MB 2025-02-15 12:48:01,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21378.66 MB 2025-02-15 12:48:01,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:48:01,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:48:01,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:48:01,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20216.63 MB 2025-02-15 12:48:01,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20467.17 MB 2025-02-15 12:48:01,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 250.54 MB 2025-02-15 12:48:01,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22911.39 MB 2025-02-15 12:48:01,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23072.87 MB 2025-02-15 12:48:01,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-15 12:48:01,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20651.20 MB 2025-02-15 12:48:01,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:48:01,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:48:01,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:01,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20625.65 MB 2025-02-15 12:48:01,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20824.77 MB 2025-02-15 12:48:01,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.12 MB 2025-02-15 12:48:01,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23072.87 MB 2025-02-15 12:48:01,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23072.87 MB 2025-02-15 12:48:01,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:01,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20824.77 MB 2025-02-15 12:48:01,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:48:01,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:48:01,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.30 seconds 2025-02-15 12:48:01,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:01,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17816.43 MB 2025-02-15 12:48:01,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21024.89 MB 2025-02-15 12:48:01,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3208.46 MB 2025-02-15 12:48:01,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41748.00 MB 2025-02-15 12:48:01,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23072.87 MB 2025-02-15 12:48:01,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18675.14 MB 2025-02-15 12:48:01,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21024.89 MB 2025-02-15 12:48:02,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:48:02,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:48:02,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:48:02,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:02,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18192.59 MB 2025-02-15 12:48:02,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18292.58 MB 2025-02-15 12:48:02,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.99 MB 2025-02-15 12:48:02,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23072.87 MB 2025-02-15 12:48:02,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23072.87 MB 2025-02-15 12:48:02,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:02,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18893.24 MB 2025-02-15 12:48:02,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 12:48:02,214 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:02,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:48:02,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:48:02,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:48:02,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:02,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18292.58 MB 2025-02-15 12:48:02,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22468.01 MB 2025-02-15 12:48:02,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.43 MB 2025-02-15 12:48:02,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23072.87 MB 2025-02-15 12:48:02,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33512.49 MB 2025-02-15 12:48:02,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-15 12:48:02,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26641.97 MB 2025-02-15 12:48:02,383 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 12:48:02,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,385 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:48:02,390 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:48:02,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,391 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:48:02,391 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:02,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,392 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,392 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:02,398 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:48:02,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,399 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,399 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:02,399 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:48:02,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,399 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:02,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,400 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:48:02,400 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:48:02,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,401 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:02,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,405 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,406 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,407 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:02,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:02,411 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:04,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:04,642 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:04,647 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:48:04,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:04,648 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 472, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:48:04,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:04,649 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 472, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:48:11,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:48:11,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:48:11,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.33 seconds 2025-02-15 12:48:11,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:11,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26000.50 MB 2025-02-15 12:48:11,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27670.88 MB 2025-02-15 12:48:11,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1670.38 MB 2025-02-15 12:48:11,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33512.49 MB 2025-02-15 12:48:11,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33355.20 MB 2025-02-15 12:48:11,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -157.29 MB 2025-02-15 12:48:11,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36604.33 MB 2025-02-15 12:48:12,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:48:12,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:48:12,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:48:12,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:12,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27670.88 MB 2025-02-15 12:48:12,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27975.50 MB 2025-02-15 12:48:12,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.62 MB 2025-02-15 12:48:12,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33355.20 MB 2025-02-15 12:48:12,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37098.62 MB 2025-02-15 12:48:12,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3743.42 MB 2025-02-15 12:48:12,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35106.24 MB 2025-02-15 12:48:13,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:48:13,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:48:13,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:48:13,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:13,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27975.50 MB 2025-02-15 12:48:13,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28506.34 MB 2025-02-15 12:48:13,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:48:13,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37098.62 MB 2025-02-15 12:48:13,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35479.62 MB 2025-02-15 12:48:13,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1619.00 MB 2025-02-15 12:48:13,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32484.89 MB 2025-02-15 12:48:13,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:48:13,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:48:13,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:48:13,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:13,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28506.34 MB 2025-02-15 12:48:13,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30395.84 MB 2025-02-15 12:48:13,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 12:48:13,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35479.62 MB 2025-02-15 12:48:13,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35479.62 MB 2025-02-15 12:48:13,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:13,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31813.26 MB 2025-02-15 12:48:14,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:48:14,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:48:14,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:48:14,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30395.84 MB 2025-02-15 12:48:14,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32637.69 MB 2025-02-15 12:48:14,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:48:14,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35479.62 MB 2025-02-15 12:48:14,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 12:48:14,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 12:48:14,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38181.97 MB 2025-02-15 12:48:14,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:48:14,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:48:14,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:48:14,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28506.34 MB 2025-02-15 12:48:14,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32637.69 MB 2025-02-15 12:48:14,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 12:48:14,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35479.62 MB 2025-02-15 12:48:14,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-15 12:48:14,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 12:48:14,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38181.97 MB 2025-02-15 12:48:14,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:48:14,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:48:14,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:48:14,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33345.48 MB 2025-02-15 12:48:14,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34112.48 MB 2025-02-15 12:48:14,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:48:14,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-15 12:48:14,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41085.30 MB 2025-02-15 12:48:14,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:48:14,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34820.27 MB 2025-02-15 12:48:14,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:48:14,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:48:14,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:48:14,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34525.37 MB 2025-02-15 12:48:14,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34731.32 MB 2025-02-15 12:48:14,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.95 MB 2025-02-15 12:48:14,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41085.30 MB 2025-02-15 12:48:14,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41085.30 MB 2025-02-15 12:48:14,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:14,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34931.98 MB 2025-02-15 12:48:14,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:48:14,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:48:14,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.68 seconds 2025-02-15 12:48:14,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24356.01 MB 2025-02-15 12:48:14,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34932.00 MB 2025-02-15 12:48:14,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10575.98 MB 2025-02-15 12:48:14,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33512.49 MB 2025-02-15 12:48:14,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41085.30 MB 2025-02-15 12:48:14,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7572.82 MB 2025-02-15 12:48:14,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34932.00 MB 2025-02-15 12:48:14,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:48:14,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:48:14,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:48:14,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34932.00 MB 2025-02-15 12:48:14,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35032.27 MB 2025-02-15 12:48:14,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.27 MB 2025-02-15 12:48:14,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41085.30 MB 2025-02-15 12:48:14,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41085.30 MB 2025-02-15 12:48:14,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:14,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35633.89 MB 2025-02-15 12:48:14,609 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 12:48:14,609 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:48:14,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:48:14,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:48:14,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:48:14,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:14,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25618.54 MB 2025-02-15 12:48:14,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29804.82 MB 2025-02-15 12:48:14,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.28 MB 2025-02-15 12:48:14,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41085.30 MB 2025-02-15 12:48:14,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49459.23 MB 2025-02-15 12:48:14,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 12:48:14,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33990.73 MB 2025-02-15 12:48:14,775 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 12:48:14,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,777 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:48:14,782 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:48:14,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,783 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:48:14,783 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:48:14,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,784 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,785 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:14,790 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:48:14,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,791 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,792 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:14,792 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:48:14,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,792 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:14,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,793 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:48:14,793 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:48:14,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,793 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:14,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,796 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,797 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,798 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:14,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:14,801 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:21,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:21,244 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:21,252 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:48:21,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:21,254 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 60, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:48:21,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:21,256 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 60, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:48:22,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:48:22,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:48:22,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.97 seconds 2025-02-15 12:48:22,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23251.17 MB 2025-02-15 12:48:22,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23463.51 MB 2025-02-15 12:48:22,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.34 MB 2025-02-15 12:48:22,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49459.23 MB 2025-02-15 12:48:22,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27088.91 MB 2025-02-15 12:48:22,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22370.32 MB 2025-02-15 12:48:22,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31918.34 MB 2025-02-15 12:48:22,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:48:22,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:48:22,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:22,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23463.51 MB 2025-02-15 12:48:22,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23566.38 MB 2025-02-15 12:48:22,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 102.88 MB 2025-02-15 12:48:22,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27088.91 MB 2025-02-15 12:48:22,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27088.91 MB 2025-02-15 12:48:22,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23884.95 MB 2025-02-15 12:48:22,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:48:22,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:48:22,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 12:48:22,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23566.38 MB 2025-02-15 12:48:22,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23646.01 MB 2025-02-15 12:48:22,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 79.63 MB 2025-02-15 12:48:22,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27088.91 MB 2025-02-15 12:48:22,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27466.40 MB 2025-02-15 12:48:22,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 377.49 MB 2025-02-15 12:48:22,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27397.75 MB 2025-02-15 12:48:22,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:48:22,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:48:22,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:22,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23645.94 MB 2025-02-15 12:48:22,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23929.31 MB 2025-02-15 12:48:22,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.36 MB 2025-02-15 12:48:22,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27466.40 MB 2025-02-15 12:48:22,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27466.40 MB 2025-02-15 12:48:22,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24141.93 MB 2025-02-15 12:48:22,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:48:22,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:48:22,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:48:22,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23929.31 MB 2025-02-15 12:48:22,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24273.50 MB 2025-02-15 12:48:22,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.20 MB 2025-02-15 12:48:22,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27466.40 MB 2025-02-15 12:48:22,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27466.40 MB 2025-02-15 12:48:22,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25097.23 MB 2025-02-15 12:48:22,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:48:22,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:48:22,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:48:22,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23645.94 MB 2025-02-15 12:48:22,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24273.50 MB 2025-02-15 12:48:22,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 627.56 MB 2025-02-15 12:48:22,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27466.40 MB 2025-02-15 12:48:22,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27466.40 MB 2025-02-15 12:48:22,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25097.23 MB 2025-02-15 12:48:22,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:48:22,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:48:22,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:48:22,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.86 MB 2025-02-15 12:48:22,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24571.40 MB 2025-02-15 12:48:22,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 144.54 MB 2025-02-15 12:48:22,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27466.40 MB 2025-02-15 12:48:22,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27554.48 MB 2025-02-15 12:48:22,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 88.08 MB 2025-02-15 12:48:22,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24677.57 MB 2025-02-15 12:48:22,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:48:22,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:48:22,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:22,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24662.83 MB 2025-02-15 12:48:22,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24791.61 MB 2025-02-15 12:48:22,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 128.77 MB 2025-02-15 12:48:22,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27554.48 MB 2025-02-15 12:48:22,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27554.48 MB 2025-02-15 12:48:22,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24791.61 MB 2025-02-15 12:48:22,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:48:22,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:48:22,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.42 seconds 2025-02-15 12:48:22,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23042.13 MB 2025-02-15 12:48:22,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24921.38 MB 2025-02-15 12:48:22,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1879.26 MB 2025-02-15 12:48:22,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49459.23 MB 2025-02-15 12:48:22,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27554.48 MB 2025-02-15 12:48:22,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21904.75 MB 2025-02-15 12:48:22,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24921.38 MB 2025-02-15 12:48:22,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:48:22,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:48:22,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:48:22,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24921.38 MB 2025-02-15 12:48:22,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24986.23 MB 2025-02-15 12:48:22,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 64.84 MB 2025-02-15 12:48:22,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27554.48 MB 2025-02-15 12:48:22,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27554.48 MB 2025-02-15 12:48:22,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:22,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25375.29 MB 2025-02-15 12:48:22,871 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5263, cut from 5265 2025-02-15 12:48:22,872 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:22,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:48:22,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:48:22,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:48:22,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:22,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24986.23 MB 2025-02-15 12:48:22,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26038.55 MB 2025-02-15 12:48:22,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1052.32 MB 2025-02-15 12:48:22,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27554.48 MB 2025-02-15 12:48:22,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32969.33 MB 2025-02-15 12:48:22,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5414.85 MB 2025-02-15 12:48:22,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29047.26 MB 2025-02-15 12:48:23,040 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5055] 2025-02-15 12:48:23,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,042 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,044 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:48:23,052 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:48:23,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,054 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:48:23,054 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:23,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,055 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,056 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:23,065 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:48:23,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,067 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,067 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:23,068 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:48:23,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,068 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:23,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,069 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:48:23,070 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:48:23,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,070 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:23,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,080 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,083 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,086 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:23,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:23,092 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:28,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:28,366 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:28,374 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:48:28,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:28,376 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 121, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:48:28,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:28,378 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 121, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:48:30,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:48:30,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:48:30,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.87 seconds 2025-02-15 12:48:30,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23797.82 MB 2025-02-15 12:48:30,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24226.04 MB 2025-02-15 12:48:30,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 428.21 MB 2025-02-15 12:48:30,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36740.01 MB 2025-02-15 12:48:30,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27051.16 MB 2025-02-15 12:48:30,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9688.84 MB 2025-02-15 12:48:30,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33042.70 MB 2025-02-15 12:48:30,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:48:30,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:48:30,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:30,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24226.04 MB 2025-02-15 12:48:30,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24433.50 MB 2025-02-15 12:48:30,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.47 MB 2025-02-15 12:48:30,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27051.16 MB 2025-02-15 12:48:30,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27051.16 MB 2025-02-15 12:48:30,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:30,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25075.89 MB 2025-02-15 12:48:30,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:48:30,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:48:30,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.59 seconds 2025-02-15 12:48:30,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24433.50 MB 2025-02-15 12:48:30,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24594.08 MB 2025-02-15 12:48:30,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 160.58 MB 2025-02-15 12:48:30,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27051.16 MB 2025-02-15 12:48:30,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27051.16 MB 2025-02-15 12:48:30,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:30,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28519.26 MB 2025-02-15 12:48:30,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:48:30,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:48:30,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:30,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24594.02 MB 2025-02-15 12:48:30,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25165.46 MB 2025-02-15 12:48:30,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 571.45 MB 2025-02-15 12:48:30,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27051.16 MB 2025-02-15 12:48:30,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27051.16 MB 2025-02-15 12:48:30,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:30,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25594.24 MB 2025-02-15 12:48:30,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:48:30,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:48:30,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:48:30,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25165.46 MB 2025-02-15 12:48:30,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25859.53 MB 2025-02-15 12:48:30,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.07 MB 2025-02-15 12:48:30,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27051.16 MB 2025-02-15 12:48:30,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28487.71 MB 2025-02-15 12:48:30,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1436.55 MB 2025-02-15 12:48:30,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27520.77 MB 2025-02-15 12:48:30,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:48:30,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:48:30,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:48:30,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:30,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24594.08 MB 2025-02-15 12:48:30,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25859.53 MB 2025-02-15 12:48:30,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1265.45 MB 2025-02-15 12:48:30,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27051.16 MB 2025-02-15 12:48:30,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28487.71 MB 2025-02-15 12:48:30,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1436.55 MB 2025-02-15 12:48:30,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27520.77 MB 2025-02-15 12:48:31,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:48:31,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:48:31,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:48:31,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:31,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26168.79 MB 2025-02-15 12:48:31,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26460.29 MB 2025-02-15 12:48:31,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.49 MB 2025-02-15 12:48:31,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28487.71 MB 2025-02-15 12:48:31,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28672.26 MB 2025-02-15 12:48:31,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-15 12:48:31,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26674.39 MB 2025-02-15 12:48:31,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:48:31,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:48:31,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:31,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:31,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26644.67 MB 2025-02-15 12:48:31,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26843.80 MB 2025-02-15 12:48:31,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.13 MB 2025-02-15 12:48:31,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28672.26 MB 2025-02-15 12:48:31,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28672.26 MB 2025-02-15 12:48:31,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:31,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26843.80 MB 2025-02-15 12:48:31,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:48:31,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:48:31,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-15 12:48:31,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:31,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23376.25 MB 2025-02-15 12:48:31,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27044.41 MB 2025-02-15 12:48:31,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3668.16 MB 2025-02-15 12:48:31,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36740.01 MB 2025-02-15 12:48:31,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28672.26 MB 2025-02-15 12:48:31,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8067.74 MB 2025-02-15 12:48:31,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27044.41 MB 2025-02-15 12:48:31,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:48:31,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:48:31,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:48:31,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:31,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27044.41 MB 2025-02-15 12:48:31,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27144.64 MB 2025-02-15 12:48:31,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.23 MB 2025-02-15 12:48:31,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28672.26 MB 2025-02-15 12:48:31,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28672.26 MB 2025-02-15 12:48:31,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:31,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27746.04 MB 2025-02-15 12:48:31,329 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 12:48:31,329 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:48:31,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:48:31,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:48:31,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:48:31,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:31,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27144.64 MB 2025-02-15 12:48:31,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31329.38 MB 2025-02-15 12:48:31,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.74 MB 2025-02-15 12:48:31,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28672.26 MB 2025-02-15 12:48:31,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-15 12:48:31,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 12:48:31,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35513.60 MB 2025-02-15 12:48:31,500 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 12:48:31,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,501 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,502 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:48:31,507 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:48:31,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,508 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:48:31,508 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:48:31,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,509 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,509 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:31,515 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:48:31,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,516 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,516 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:31,516 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:48:31,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,517 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:31,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,518 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:48:31,518 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:48:31,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,519 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:31,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,522 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,523 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,523 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:31,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:31,527 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:50,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:50,175 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:50,180 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:48:50,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:50,181 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 81, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:48:50,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:50,182 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 81, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:48:51,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:48:51,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:48:51,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.28 seconds 2025-02-15 12:48:51,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23640.74 MB 2025-02-15 12:48:51,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23927.40 MB 2025-02-15 12:48:51,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 286.65 MB 2025-02-15 12:48:51,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 12:48:51,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11760.83 MB 2025-02-15 12:48:51,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32886.67 MB 2025-02-15 12:48:51,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:48:51,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:48:51,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:51,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23927.40 MB 2025-02-15 12:48:51,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24066.28 MB 2025-02-15 12:48:51,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 138.88 MB 2025-02-15 12:48:51,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:51,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:51,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24496.32 MB 2025-02-15 12:48:51,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:48:51,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:48:51,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.40 seconds 2025-02-15 12:48:51,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24066.28 MB 2025-02-15 12:48:51,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24173.77 MB 2025-02-15 12:48:51,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 107.50 MB 2025-02-15 12:48:51,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:51,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:51,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28152.03 MB 2025-02-15 12:48:51,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:48:51,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:48:51,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:51,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24173.70 MB 2025-02-15 12:48:51,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24556.24 MB 2025-02-15 12:48:51,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 382.54 MB 2025-02-15 12:48:51,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:51,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:51,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24843.27 MB 2025-02-15 12:48:51,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:48:51,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:48:51,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:48:51,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24556.24 MB 2025-02-15 12:48:51,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.67 MB 2025-02-15 12:48:51,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 465.44 MB 2025-02-15 12:48:51,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:51,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:51,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26133.72 MB 2025-02-15 12:48:51,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:48:51,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:48:51,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:48:51,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:51,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24173.77 MB 2025-02-15 12:48:51,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.67 MB 2025-02-15 12:48:51,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 847.90 MB 2025-02-15 12:48:51,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:51,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27376.22 MB 2025-02-15 12:48:51,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:51,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26133.72 MB 2025-02-15 12:48:52,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:48:52,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:48:52,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 12:48:52,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:52,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25228.70 MB 2025-02-15 12:48:52,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25423.83 MB 2025-02-15 12:48:52,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.13 MB 2025-02-15 12:48:52,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27376.22 MB 2025-02-15 12:48:52,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27497.86 MB 2025-02-15 12:48:52,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 121.63 MB 2025-02-15 12:48:52,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25567.16 MB 2025-02-15 12:48:52,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:48:52,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:48:52,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:48:52,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:52,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25547.27 MB 2025-02-15 12:48:52,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25719.51 MB 2025-02-15 12:48:52,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.25 MB 2025-02-15 12:48:52,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27497.86 MB 2025-02-15 12:48:52,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27497.86 MB 2025-02-15 12:48:52,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:52,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25719.51 MB 2025-02-15 12:48:52,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:48:52,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:48:52,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.82 seconds 2025-02-15 12:48:52,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:52,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23358.53 MB 2025-02-15 12:48:52,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25892.67 MB 2025-02-15 12:48:52,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2534.14 MB 2025-02-15 12:48:52,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39137.05 MB 2025-02-15 12:48:52,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27497.86 MB 2025-02-15 12:48:52,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11639.19 MB 2025-02-15 12:48:52,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.67 MB 2025-02-15 12:48:52,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:48:52,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:48:52,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:48:52,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:52,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25892.67 MB 2025-02-15 12:48:52,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23746.65 MB 2025-02-15 12:48:52,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2146.02 MB 2025-02-15 12:48:52,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27497.86 MB 2025-02-15 12:48:52,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27497.86 MB 2025-02-15 12:48:52,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:48:52,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.67 MB 2025-02-15 12:48:52,244 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7027, cut from 7029 2025-02-15 12:48:52,244 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:52,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:48:52,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:48:52,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:48:52,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:48:52,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23746.65 MB 2025-02-15 12:48:52,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27358.86 MB 2025-02-15 12:48:52,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3612.20 MB 2025-02-15 12:48:52,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27497.86 MB 2025-02-15 12:48:52,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36532.39 MB 2025-02-15 12:48:52,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9034.53 MB 2025-02-15 12:48:52,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30970.55 MB 2025-02-15 12:48:52,392 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6819] 2025-02-15 12:48:52,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,394 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,394 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:48:52,399 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:48:52,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,400 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:48:52,400 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:48:52,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,401 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,402 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:52,407 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:48:52,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,408 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,409 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:52,409 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:48:52,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,409 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:52,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,410 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:48:52,410 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:48:52,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,410 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:48:52,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,415 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,416 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,418 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:48:52,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:48:52,422 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:02,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:02,730 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:02,735 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:49:02,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:02,736 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 339, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:49:02,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:02,737 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 339, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:49:08,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:49:08,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:49:08,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.29 seconds 2025-02-15 12:49:08,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:08,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25561.65 MB 2025-02-15 12:49:08,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26761.35 MB 2025-02-15 12:49:08,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1199.70 MB 2025-02-15 12:49:08,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40546.34 MB 2025-02-15 12:49:08,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29926.36 MB 2025-02-15 12:49:08,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10619.98 MB 2025-02-15 12:49:08,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35713.30 MB 2025-02-15 12:49:08,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:49:08,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:49:08,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:49:08,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:08,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26761.35 MB 2025-02-15 12:49:08,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26984.36 MB 2025-02-15 12:49:08,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.01 MB 2025-02-15 12:49:08,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29926.36 MB 2025-02-15 12:49:08,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32954.65 MB 2025-02-15 12:49:08,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3028.29 MB 2025-02-15 12:49:08,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30806.61 MB 2025-02-15 12:49:09,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:49:09,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:49:09,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.38 seconds 2025-02-15 12:49:09,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26984.36 MB 2025-02-15 12:49:09,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27366.57 MB 2025-02-15 12:49:09,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 382.21 MB 2025-02-15 12:49:09,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32954.65 MB 2025-02-15 12:49:09,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30238.83 MB 2025-02-15 12:49:09,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2715.81 MB 2025-02-15 12:49:09,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31324.92 MB 2025-02-15 12:49:09,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:49:09,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:49:09,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:49:09,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27366.57 MB 2025-02-15 12:49:09,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28727.11 MB 2025-02-15 12:49:09,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1360.54 MB 2025-02-15 12:49:09,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30238.83 MB 2025-02-15 12:49:09,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32277.27 MB 2025-02-15 12:49:09,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2038.43 MB 2025-02-15 12:49:09,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29747.66 MB 2025-02-15 12:49:09,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:49:09,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:49:09,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:49:09,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28727.11 MB 2025-02-15 12:49:09,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.26 MB 2025-02-15 12:49:09,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1614.15 MB 2025-02-15 12:49:09,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32277.27 MB 2025-02-15 12:49:09,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 12:49:09,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4076.86 MB 2025-02-15 12:49:09,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.13 MB 2025-02-15 12:49:09,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:49:09,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:49:09,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:49:09,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27366.57 MB 2025-02-15 12:49:09,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.26 MB 2025-02-15 12:49:09,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2974.70 MB 2025-02-15 12:49:09,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30238.83 MB 2025-02-15 12:49:09,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-15 12:49:09,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6115.30 MB 2025-02-15 12:49:09,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.13 MB 2025-02-15 12:49:09,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:49:09,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:49:09,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:49:09,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30850.87 MB 2025-02-15 12:49:09,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31403.11 MB 2025-02-15 12:49:09,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.24 MB 2025-02-15 12:49:09,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36354.13 MB 2025-02-15 12:49:09,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36651.93 MB 2025-02-15 12:49:09,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 297.80 MB 2025-02-15 12:49:09,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31912.72 MB 2025-02-15 12:49:09,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:49:09,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:49:09,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:49:09,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31700.40 MB 2025-02-15 12:49:09,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31908.19 MB 2025-02-15 12:49:09,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.80 MB 2025-02-15 12:49:09,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36651.93 MB 2025-02-15 12:49:09,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36651.93 MB 2025-02-15 12:49:09,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:49:09,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32024.51 MB 2025-02-15 12:49:09,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:49:09,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:49:09,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.02 seconds 2025-02-15 12:49:09,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:09,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24380.54 MB 2025-02-15 12:49:09,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32109.27 MB 2025-02-15 12:49:09,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7728.72 MB 2025-02-15 12:49:09,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40546.34 MB 2025-02-15 12:49:09,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36651.93 MB 2025-02-15 12:49:09,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3894.41 MB 2025-02-15 12:49:09,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32109.27 MB 2025-02-15 12:49:10,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:49:10,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:49:10,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:49:10,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:10,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32109.27 MB 2025-02-15 12:49:10,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32209.73 MB 2025-02-15 12:49:10,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:49:10,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36651.93 MB 2025-02-15 12:49:10,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36651.93 MB 2025-02-15 12:49:10,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:49:10,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32812.53 MB 2025-02-15 12:49:10,043 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:49:10,043 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:49:10,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:49:10,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:49:10,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:49:10,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:10,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.73 MB 2025-02-15 12:49:10,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29540.61 MB 2025-02-15 12:49:10,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2669.12 MB 2025-02-15 12:49:10,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36651.93 MB 2025-02-15 12:49:10,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47141.88 MB 2025-02-15 12:49:10,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:49:10,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33734.92 MB 2025-02-15 12:49:10,208 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:49:10,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,210 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,210 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:49:10,215 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:49:10,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,216 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:49:10,216 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:49:10,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,217 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,217 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:10,223 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:49:10,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,224 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,224 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:10,224 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:49:10,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,225 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:10,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,226 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:49:10,226 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:49:10,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,227 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:10,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,230 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,231 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,232 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:10,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:10,236 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:37,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:37,486 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:37,491 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:49:37,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:37,492 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 240, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:49:37,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:37,493 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 240, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:49:41,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:49:41,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:49:41,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.70 seconds 2025-02-15 12:49:41,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:41,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24993.47 MB 2025-02-15 12:49:41,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25842.82 MB 2025-02-15 12:49:41,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 849.35 MB 2025-02-15 12:49:41,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47141.88 MB 2025-02-15 12:49:41,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28271.71 MB 2025-02-15 12:49:41,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18870.17 MB 2025-02-15 12:49:41,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34692.14 MB 2025-02-15 12:49:41,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:49:41,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:49:41,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:49:41,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:41,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25842.82 MB 2025-02-15 12:49:41,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25945.83 MB 2025-02-15 12:49:41,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 103.02 MB 2025-02-15 12:49:41,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28271.71 MB 2025-02-15 12:49:41,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30683.43 MB 2025-02-15 12:49:41,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2411.72 MB 2025-02-15 12:49:41,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28596.95 MB 2025-02-15 12:49:42,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:49:42,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:49:42,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-15 12:49:42,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25945.83 MB 2025-02-15 12:49:42,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26205.95 MB 2025-02-15 12:49:42,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 260.11 MB 2025-02-15 12:49:42,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30683.43 MB 2025-02-15 12:49:42,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29651.63 MB 2025-02-15 12:49:42,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1031.80 MB 2025-02-15 12:49:42,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30201.46 MB 2025-02-15 12:49:42,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:49:42,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:49:42,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:49:42,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26205.88 MB 2025-02-15 12:49:42,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27131.53 MB 2025-02-15 12:49:42,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 925.65 MB 2025-02-15 12:49:42,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29651.63 MB 2025-02-15 12:49:42,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29651.63 MB 2025-02-15 12:49:42,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:49:42,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27826.07 MB 2025-02-15 12:49:42,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:49:42,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:49:42,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:49:42,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27131.53 MB 2025-02-15 12:49:42,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28230.07 MB 2025-02-15 12:49:42,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1098.54 MB 2025-02-15 12:49:42,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29651.63 MB 2025-02-15 12:49:42,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32201.77 MB 2025-02-15 12:49:42,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2550.14 MB 2025-02-15 12:49:42,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30950.93 MB 2025-02-15 12:49:42,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:49:42,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:49:42,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:49:42,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26205.88 MB 2025-02-15 12:49:42,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28230.07 MB 2025-02-15 12:49:42,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2024.19 MB 2025-02-15 12:49:42,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29651.63 MB 2025-02-15 12:49:42,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32201.77 MB 2025-02-15 12:49:42,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2550.14 MB 2025-02-15 12:49:42,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30950.93 MB 2025-02-15 12:49:42,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:49:42,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:49:42,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:49:42,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28576.89 MB 2025-02-15 12:49:42,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28952.72 MB 2025-02-15 12:49:42,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 375.83 MB 2025-02-15 12:49:42,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32201.77 MB 2025-02-15 12:49:42,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 12:49:42,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-15 12:49:42,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29302.61 MB 2025-02-15 12:49:42,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:49:42,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:49:42,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:49:42,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29155.04 MB 2025-02-15 12:49:42,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29357.03 MB 2025-02-15 12:49:42,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.99 MB 2025-02-15 12:49:42,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 12:49:42,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 12:49:42,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:49:42,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29387.08 MB 2025-02-15 12:49:42,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:49:42,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:49:42,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.88 seconds 2025-02-15 12:49:42,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24157.29 MB 2025-02-15 12:49:42,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29557.91 MB 2025-02-15 12:49:42,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5400.62 MB 2025-02-15 12:49:42,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47141.88 MB 2025-02-15 12:49:42,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 12:49:42,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14738.78 MB 2025-02-15 12:49:42,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29557.91 MB 2025-02-15 12:49:42,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:49:42,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:49:42,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:49:42,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29557.91 MB 2025-02-15 12:49:42,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.44 MB 2025-02-15 12:49:42,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4679.47 MB 2025-02-15 12:49:42,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 12:49:42,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 12:49:42,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:49:42,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29792.77 MB 2025-02-15 12:49:42,658 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 12:49:42,658 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:49:42,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:49:42,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:49:42,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:49:42,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:49:42,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.44 MB 2025-02-15 12:49:42,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29068.82 MB 2025-02-15 12:49:42,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 12:49:42,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 12:49:42,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42882.56 MB 2025-02-15 12:49:42,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-15 12:49:42,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33258.93 MB 2025-02-15 12:49:42,827 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 12:49:42,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,830 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:49:42,835 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:49:42,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,836 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:49:42,837 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:49:42,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,837 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,838 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:42,844 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:49:42,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,844 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,845 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:42,845 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:49:42,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,845 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:42,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,846 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:49:42,846 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:49:42,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,846 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:42,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,854 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,856 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,858 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:49:42,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:42,863 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:52,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:52,662 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:49:52,667 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:49:52,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:52,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 680, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:49:52,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:49:52,669 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 680, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:50:03,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:50:03,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:50:03,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.52 seconds 2025-02-15 12:50:03,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:03,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28180.64 MB 2025-02-15 12:50:03,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30588.17 MB 2025-02-15 12:50:03,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2407.53 MB 2025-02-15 12:50:03,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47139.78 MB 2025-02-15 12:50:03,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35855.01 MB 2025-02-15 12:50:03,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11284.77 MB 2025-02-15 12:50:03,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39463.95 MB 2025-02-15 12:50:03,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:50:03,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:50:03,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:50:03,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:03,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30588.17 MB 2025-02-15 12:50:03,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29786.54 MB 2025-02-15 12:50:03,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -801.63 MB 2025-02-15 12:50:03,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35855.01 MB 2025-02-15 12:50:03,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41924.17 MB 2025-02-15 12:50:03,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6069.16 MB 2025-02-15 12:50:03,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39463.11 MB 2025-02-15 12:50:05,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:50:05,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:50:05,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:50:05,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29786.54 MB 2025-02-15 12:50:05,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30317.38 MB 2025-02-15 12:50:05,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:50:05,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41924.17 MB 2025-02-15 12:50:05,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34863.05 MB 2025-02-15 12:50:05,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7061.11 MB 2025-02-15 12:50:05,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34295.93 MB 2025-02-15 12:50:05,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:50:05,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:50:05,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:05,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30317.38 MB 2025-02-15 12:50:05,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32206.50 MB 2025-02-15 12:50:05,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.12 MB 2025-02-15 12:50:05,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34863.05 MB 2025-02-15 12:50:05,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37694.21 MB 2025-02-15 12:50:05,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 12:50:05,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33623.93 MB 2025-02-15 12:50:05,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:50:05,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:50:05,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:50:05,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32206.50 MB 2025-02-15 12:50:05,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34448.36 MB 2025-02-15 12:50:05,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:50:05,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37694.21 MB 2025-02-15 12:50:05,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43356.52 MB 2025-02-15 12:50:05,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:50:05,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39992.64 MB 2025-02-15 12:50:05,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:50:05,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:50:05,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:50:05,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30317.38 MB 2025-02-15 12:50:05,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34448.36 MB 2025-02-15 12:50:05,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.98 MB 2025-02-15 12:50:05,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34863.05 MB 2025-02-15 12:50:05,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43356.52 MB 2025-02-15 12:50:05,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 12:50:05,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39992.64 MB 2025-02-15 12:50:05,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:50:05,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:50:05,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:50:05,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35156.15 MB 2025-02-15 12:50:05,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35923.15 MB 2025-02-15 12:50:05,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:50:05,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43356.52 MB 2025-02-15 12:50:05,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43771.76 MB 2025-02-15 12:50:05,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:50:05,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36630.94 MB 2025-02-15 12:50:05,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:50:05,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:50:05,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:05,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36336.04 MB 2025-02-15 12:50:05,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36542.05 MB 2025-02-15 12:50:05,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.02 MB 2025-02-15 12:50:05,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43771.76 MB 2025-02-15 12:50:05,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43771.76 MB 2025-02-15 12:50:05,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:05,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36752.13 MB 2025-02-15 12:50:05,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:50:05,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:50:05,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.90 seconds 2025-02-15 12:50:05,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25811.46 MB 2025-02-15 12:50:05,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36743.13 MB 2025-02-15 12:50:05,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10931.66 MB 2025-02-15 12:50:05,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47139.78 MB 2025-02-15 12:50:05,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43771.76 MB 2025-02-15 12:50:05,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3368.03 MB 2025-02-15 12:50:05,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36752.13 MB 2025-02-15 12:50:05,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:50:05,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:50:05,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:50:05,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36743.13 MB 2025-02-15 12:50:05,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36843.59 MB 2025-02-15 12:50:05,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:50:05,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43771.76 MB 2025-02-15 12:50:05,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43771.76 MB 2025-02-15 12:50:05,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:05,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37446.39 MB 2025-02-15 12:50:05,857 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:50:05,857 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:50:05,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:50:05,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:50:05,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:05,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:05,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27074.38 MB 2025-02-15 12:50:05,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31268.87 MB 2025-02-15 12:50:05,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:50:05,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43771.76 MB 2025-02-15 12:50:05,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52162.46 MB 2025-02-15 12:50:05,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:50:05,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35463.17 MB 2025-02-15 12:50:06,023 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:50:06,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,025 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:50:06,030 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:50:06,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,031 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:50:06,031 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 12:50:06,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,032 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,032 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:06,038 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:50:06,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,039 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,039 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:06,039 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:50:06,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,040 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:06,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,040 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:50:06,040 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:50:06,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,041 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:06,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,044 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,045 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,046 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:06,050 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:06,051 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:13,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:13,706 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:13,714 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:50:13,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:13,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 210, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:50:13,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:13,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 210, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:50:17,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:50:17,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:50:17,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-15 12:50:17,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:17,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25026.88 MB 2025-02-15 12:50:17,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25770.06 MB 2025-02-15 12:50:17,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 743.18 MB 2025-02-15 12:50:17,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56541.32 MB 2025-02-15 12:50:17,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28225.57 MB 2025-02-15 12:50:17,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28315.75 MB 2025-02-15 12:50:17,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34725.55 MB 2025-02-15 12:50:17,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:50:17,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:50:17,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:17,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:17,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25770.06 MB 2025-02-15 12:50:17,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26130.70 MB 2025-02-15 12:50:17,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 360.64 MB 2025-02-15 12:50:17,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28225.57 MB 2025-02-15 12:50:17,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30435.97 MB 2025-02-15 12:50:17,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2210.40 MB 2025-02-15 12:50:17,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28721.01 MB 2025-02-15 12:50:18,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:50:18,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:50:18,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-15 12:50:18,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26130.70 MB 2025-02-15 12:50:18,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26409.40 MB 2025-02-15 12:50:18,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.69 MB 2025-02-15 12:50:18,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30435.97 MB 2025-02-15 12:50:18,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28858.91 MB 2025-02-15 12:50:18,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1577.06 MB 2025-02-15 12:50:18,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30386.33 MB 2025-02-15 12:50:18,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:50:18,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:50:18,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:18,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26409.40 MB 2025-02-15 12:50:18,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27401.16 MB 2025-02-15 12:50:18,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 991.76 MB 2025-02-15 12:50:18,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28858.91 MB 2025-02-15 12:50:18,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29355.93 MB 2025-02-15 12:50:18,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 497.03 MB 2025-02-15 12:50:18,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28145.31 MB 2025-02-15 12:50:18,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:50:18,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:50:18,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:50:18,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27401.16 MB 2025-02-15 12:50:18,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28578.17 MB 2025-02-15 12:50:18,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1177.01 MB 2025-02-15 12:50:18,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29355.93 MB 2025-02-15 12:50:18,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-15 12:50:18,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3231.71 MB 2025-02-15 12:50:18,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31488.88 MB 2025-02-15 12:50:18,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:50:18,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:50:18,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:50:18,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26409.40 MB 2025-02-15 12:50:18,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28578.17 MB 2025-02-15 12:50:18,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2168.77 MB 2025-02-15 12:50:18,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28858.91 MB 2025-02-15 12:50:18,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-15 12:50:18,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3728.74 MB 2025-02-15 12:50:18,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31488.88 MB 2025-02-15 12:50:18,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:50:18,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:50:18,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 12:50:18,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28949.76 MB 2025-02-15 12:50:18,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29352.43 MB 2025-02-15 12:50:18,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.68 MB 2025-02-15 12:50:18,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32587.64 MB 2025-02-15 12:50:18,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32803.65 MB 2025-02-15 12:50:18,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-15 12:50:18,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29727.01 MB 2025-02-15 12:50:18,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:50:18,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:50:18,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:18,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29569.20 MB 2025-02-15 12:50:18,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29770.54 MB 2025-02-15 12:50:18,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.34 MB 2025-02-15 12:50:18,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32803.65 MB 2025-02-15 12:50:18,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-15 12:50:18,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:50:18,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29808.89 MB 2025-02-15 12:50:18,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:50:18,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:50:18,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.80 seconds 2025-02-15 12:50:18,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24295.22 MB 2025-02-15 12:50:18,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29971.30 MB 2025-02-15 12:50:18,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.07 MB 2025-02-15 12:50:18,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56541.32 MB 2025-02-15 12:50:18,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-15 12:50:18,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23733.47 MB 2025-02-15 12:50:18,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29971.30 MB 2025-02-15 12:50:18,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:50:18,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:50:18,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 12:50:18,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24953.11 MB 2025-02-15 12:50:18,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25053.42 MB 2025-02-15 12:50:18,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.31 MB 2025-02-15 12:50:18,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32807.85 MB 2025-02-15 12:50:18,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-15 12:50:18,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:18,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25655.68 MB 2025-02-15 12:50:18,822 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 12:50:18,822 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 12:50:18,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:50:18,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:50:18,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:18,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:18,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25053.42 MB 2025-02-15 12:50:18,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29241.43 MB 2025-02-15 12:50:18,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.01 MB 2025-02-15 12:50:18,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32807.85 MB 2025-02-15 12:50:18,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43278.93 MB 2025-02-15 12:50:18,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 12:50:18,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33429.44 MB 2025-02-15 12:50:19,076 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 12:50:19,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,078 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:50:19,083 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:50:19,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,084 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:50:19,085 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 12:50:19,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,085 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,086 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:19,092 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:50:19,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,093 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,093 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:19,094 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:50:19,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,094 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:19,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,094 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:50:19,095 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:50:19,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,095 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:19,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,102 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,104 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,106 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:19,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:19,112 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:31,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:31,243 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:31,248 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:50:31,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:31,250 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 132, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:50:31,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:31,250 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 132, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:50:33,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:50:33,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:50:33,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.08 seconds 2025-02-15 12:50:33,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24605.69 MB 2025-02-15 12:50:33,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25072.83 MB 2025-02-15 12:50:33,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 467.14 MB 2025-02-15 12:50:33,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47779.41 MB 2025-02-15 12:50:33,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28150.07 MB 2025-02-15 12:50:33,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19629.34 MB 2025-02-15 12:50:33,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34077.87 MB 2025-02-15 12:50:33,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:50:33,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:50:33,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:33,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25072.83 MB 2025-02-15 12:50:33,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25130.61 MB 2025-02-15 12:50:33,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 57.78 MB 2025-02-15 12:50:33,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28150.07 MB 2025-02-15 12:50:33,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28150.07 MB 2025-02-15 12:50:33,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:33,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26589.87 MB 2025-02-15 12:50:33,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:50:33,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:50:33,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-15 12:50:33,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25130.61 MB 2025-02-15 12:50:33,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25273.94 MB 2025-02-15 12:50:33,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 143.33 MB 2025-02-15 12:50:33,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28150.07 MB 2025-02-15 12:50:33,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28177.33 MB 2025-02-15 12:50:33,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27.26 MB 2025-02-15 12:50:33,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29216.36 MB 2025-02-15 12:50:33,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:50:33,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:50:33,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:50:33,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25273.87 MB 2025-02-15 12:50:33,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25783.92 MB 2025-02-15 12:50:33,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 510.05 MB 2025-02-15 12:50:33,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28177.33 MB 2025-02-15 12:50:33,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28177.33 MB 2025-02-15 12:50:33,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:33,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.63 MB 2025-02-15 12:50:33,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:50:33,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:50:33,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:50:33,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25783.92 MB 2025-02-15 12:50:33,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26403.43 MB 2025-02-15 12:50:33,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.50 MB 2025-02-15 12:50:33,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28177.33 MB 2025-02-15 12:50:33,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28944.89 MB 2025-02-15 12:50:33,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 767.56 MB 2025-02-15 12:50:33,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27889.33 MB 2025-02-15 12:50:33,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:50:33,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:50:33,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:50:33,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:33,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25273.87 MB 2025-02-15 12:50:33,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26403.43 MB 2025-02-15 12:50:33,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1129.55 MB 2025-02-15 12:50:33,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28177.33 MB 2025-02-15 12:50:33,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28944.89 MB 2025-02-15 12:50:33,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 767.56 MB 2025-02-15 12:50:33,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27889.33 MB 2025-02-15 12:50:34,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:50:34,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:50:34,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:50:34,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:34,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26679.46 MB 2025-02-15 12:50:34,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26939.64 MB 2025-02-15 12:50:34,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 260.17 MB 2025-02-15 12:50:34,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28944.89 MB 2025-02-15 12:50:34,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29108.47 MB 2025-02-15 12:50:34,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 12:50:34,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27130.74 MB 2025-02-15 12:50:34,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:50:34,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:50:34,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:50:34,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:34,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27104.21 MB 2025-02-15 12:50:34,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27303.83 MB 2025-02-15 12:50:34,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.62 MB 2025-02-15 12:50:34,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29108.47 MB 2025-02-15 12:50:34,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29108.47 MB 2025-02-15 12:50:34,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:34,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27303.83 MB 2025-02-15 12:50:34,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:50:34,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:50:34,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-15 12:50:34,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:34,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24145.79 MB 2025-02-15 12:50:34,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27504.63 MB 2025-02-15 12:50:34,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3358.84 MB 2025-02-15 12:50:34,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47779.41 MB 2025-02-15 12:50:34,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29108.47 MB 2025-02-15 12:50:34,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18670.94 MB 2025-02-15 12:50:34,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27504.63 MB 2025-02-15 12:50:34,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:50:34,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:50:34,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:50:34,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:34,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24532.92 MB 2025-02-15 12:50:34,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24633.25 MB 2025-02-15 12:50:34,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.33 MB 2025-02-15 12:50:34,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29108.47 MB 2025-02-15 12:50:34,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29108.47 MB 2025-02-15 12:50:34,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:34,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25235.63 MB 2025-02-15 12:50:34,325 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 12:50:34,325 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:50:34,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:50:34,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:50:34,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:34,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:34,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24633.25 MB 2025-02-15 12:50:34,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28822.09 MB 2025-02-15 12:50:34,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.84 MB 2025-02-15 12:50:34,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29108.47 MB 2025-02-15 12:50:34,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39583.74 MB 2025-02-15 12:50:34,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 12:50:34,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33010.42 MB 2025-02-15 12:50:34,493 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 12:50:34,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,494 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,495 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:50:34,500 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:50:34,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,501 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:50:34,501 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:50:34,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,502 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,502 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:34,508 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:50:34,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,509 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,509 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:34,509 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:50:34,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,510 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:34,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,510 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:50:34,510 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:50:34,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,511 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:34,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,517 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,519 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,521 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:34,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:34,526 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:51,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:51,262 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:51,266 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:50:51,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:51,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:50:51,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:51,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:50:55,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:50:55,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:50:55,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.39 seconds 2025-02-15 12:50:55,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:55,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25780.22 MB 2025-02-15 12:50:55,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26782.66 MB 2025-02-15 12:50:55,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1002.44 MB 2025-02-15 12:50:55,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44205.87 MB 2025-02-15 12:50:55,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29026.68 MB 2025-02-15 12:50:55,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15179.19 MB 2025-02-15 12:50:55,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35704.58 MB 2025-02-15 12:50:55,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:50:55,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:50:55,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:55,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:55,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26782.66 MB 2025-02-15 12:50:55,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27239.22 MB 2025-02-15 12:50:55,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 456.55 MB 2025-02-15 12:50:55,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29026.68 MB 2025-02-15 12:50:55,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32457.62 MB 2025-02-15 12:50:55,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3430.94 MB 2025-02-15 12:50:55,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30702.30 MB 2025-02-15 12:50:57,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:50:57,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:50:57,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.34 seconds 2025-02-15 12:50:57,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27239.22 MB 2025-02-15 12:50:57,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27609.48 MB 2025-02-15 12:50:57,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.26 MB 2025-02-15 12:50:57,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32457.62 MB 2025-02-15 12:50:57,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-15 12:50:57,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2965.37 MB 2025-02-15 12:50:57,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31579.77 MB 2025-02-15 12:50:57,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:50:57,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:50:57,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:57,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27609.48 MB 2025-02-15 12:50:57,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28928.32 MB 2025-02-15 12:50:57,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1318.84 MB 2025-02-15 12:50:57,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-15 12:50:57,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31797.02 MB 2025-02-15 12:50:57,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-15 12:50:57,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29917.37 MB 2025-02-15 12:50:57,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:50:57,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:50:57,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:50:57,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28928.32 MB 2025-02-15 12:50:57,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30492.56 MB 2025-02-15 12:50:57,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1564.24 MB 2025-02-15 12:50:57,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31797.02 MB 2025-02-15 12:50:57,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-15 12:50:57,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4280.29 MB 2025-02-15 12:50:57,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34360.98 MB 2025-02-15 12:50:57,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:50:57,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:50:57,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:50:57,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27609.48 MB 2025-02-15 12:50:57,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30492.56 MB 2025-02-15 12:50:57,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2883.08 MB 2025-02-15 12:50:57,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-15 12:50:57,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-15 12:50:57,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6585.06 MB 2025-02-15 12:50:57,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34360.98 MB 2025-02-15 12:50:57,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:50:57,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:50:57,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:50:57,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30986.24 MB 2025-02-15 12:50:57,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31521.48 MB 2025-02-15 12:50:57,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 535.25 MB 2025-02-15 12:50:57,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36077.31 MB 2025-02-15 12:50:57,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36364.62 MB 2025-02-15 12:50:57,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 287.31 MB 2025-02-15 12:50:57,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32015.17 MB 2025-02-15 12:50:57,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:50:57,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:50:57,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:50:57,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31809.48 MB 2025-02-15 12:50:57,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32014.90 MB 2025-02-15 12:50:57,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.42 MB 2025-02-15 12:50:57,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36364.62 MB 2025-02-15 12:50:57,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36364.62 MB 2025-02-15 12:50:57,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:57,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.11 MB 2025-02-15 12:50:57,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:50:57,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:50:57,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.05 seconds 2025-02-15 12:50:57,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24794.23 MB 2025-02-15 12:50:57,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32215.16 MB 2025-02-15 12:50:57,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7420.94 MB 2025-02-15 12:50:57,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44205.87 MB 2025-02-15 12:50:57,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36364.62 MB 2025-02-15 12:50:57,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7841.25 MB 2025-02-15 12:50:57,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32215.16 MB 2025-02-15 12:50:57,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:50:57,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:50:57,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:50:57,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32215.16 MB 2025-02-15 12:50:57,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32315.23 MB 2025-02-15 12:50:57,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.06 MB 2025-02-15 12:50:57,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36364.62 MB 2025-02-15 12:50:57,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36364.62 MB 2025-02-15 12:50:57,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:50:57,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32916.05 MB 2025-02-15 12:50:57,600 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 12:50:57,600 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 12:50:57,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:50:57,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:50:57,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:50:57,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:50:57,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32315.23 MB 2025-02-15 12:50:57,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29912.93 MB 2025-02-15 12:50:57,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2402.30 MB 2025-02-15 12:50:57,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36364.62 MB 2025-02-15 12:50:57,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46812.63 MB 2025-02-15 12:50:57,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10448.01 MB 2025-02-15 12:50:57,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34090.46 MB 2025-02-15 12:50:57,767 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 12:50:57,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,769 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:50:57,774 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:50:57,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,775 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:50:57,775 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 12:50:57,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,776 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,777 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:57,783 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:50:57,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,783 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,784 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:57,784 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:50:57,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,784 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:57,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,785 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:50:57,785 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:50:57,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,785 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:50:57,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,790 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,792 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,794 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:50:57,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:50:57,799 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:30,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:30,712 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:30,717 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:51:30,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:30,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 384, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:51:30,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:30,719 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 384, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:51:36,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:51:36,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:51:36,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.92 seconds 2025-02-15 12:51:36,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:36,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26605.41 MB 2025-02-15 12:51:36,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27964.37 MB 2025-02-15 12:51:36,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1358.95 MB 2025-02-15 12:51:36,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51556.38 MB 2025-02-15 12:51:36,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30601.64 MB 2025-02-15 12:51:36,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20954.74 MB 2025-02-15 12:51:36,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36983.56 MB 2025-02-15 12:51:36,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:51:36,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:51:36,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:51:36,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:36,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27964.37 MB 2025-02-15 12:51:36,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28195.29 MB 2025-02-15 12:51:36,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.92 MB 2025-02-15 12:51:36,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30601.64 MB 2025-02-15 12:51:36,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35148.27 MB 2025-02-15 12:51:36,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4546.63 MB 2025-02-15 12:51:36,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32503.21 MB 2025-02-15 12:51:38,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:51:38,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:51:38,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.56 seconds 2025-02-15 12:51:38,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28195.29 MB 2025-02-15 12:51:38,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28623.95 MB 2025-02-15 12:51:38,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 428.65 MB 2025-02-15 12:51:38,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35148.27 MB 2025-02-15 12:51:38,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31237.08 MB 2025-02-15 12:51:38,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3911.19 MB 2025-02-15 12:51:38,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32620.78 MB 2025-02-15 12:51:38,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:51:38,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:51:38,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:51:38,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28623.95 MB 2025-02-15 12:51:38,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30149.93 MB 2025-02-15 12:51:38,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1525.98 MB 2025-02-15 12:51:38,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31237.08 MB 2025-02-15 12:51:38,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33527.17 MB 2025-02-15 12:51:38,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-15 12:51:38,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31294.50 MB 2025-02-15 12:51:38,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:51:38,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:51:38,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 12:51:38,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30149.93 MB 2025-02-15 12:51:38,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31960.24 MB 2025-02-15 12:51:38,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1810.31 MB 2025-02-15 12:51:38,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33527.17 MB 2025-02-15 12:51:38,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38107.35 MB 2025-02-15 12:51:38,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4580.18 MB 2025-02-15 12:51:38,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36438.55 MB 2025-02-15 12:51:38,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:51:38,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:51:38,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 12:51:38,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28623.95 MB 2025-02-15 12:51:38,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31960.24 MB 2025-02-15 12:51:38,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3336.29 MB 2025-02-15 12:51:38,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31237.08 MB 2025-02-15 12:51:38,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38107.35 MB 2025-02-15 12:51:38,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6870.27 MB 2025-02-15 12:51:38,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36438.55 MB 2025-02-15 12:51:38,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:51:38,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:51:38,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 12:51:38,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32531.78 MB 2025-02-15 12:51:38,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33152.44 MB 2025-02-15 12:51:38,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 620.66 MB 2025-02-15 12:51:38,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38107.35 MB 2025-02-15 12:51:38,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38440.80 MB 2025-02-15 12:51:38,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 333.45 MB 2025-02-15 12:51:38,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33723.98 MB 2025-02-15 12:51:38,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:51:38,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:51:38,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:51:38,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33485.85 MB 2025-02-15 12:51:38,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33688.99 MB 2025-02-15 12:51:38,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.13 MB 2025-02-15 12:51:38,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38440.80 MB 2025-02-15 12:51:38,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:51:38,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 12:51:38,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33801.05 MB 2025-02-15 12:51:38,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:51:38,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:51:38,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.83 seconds 2025-02-15 12:51:38,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25267.43 MB 2025-02-15 12:51:38,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33890.06 MB 2025-02-15 12:51:38,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8622.63 MB 2025-02-15 12:51:38,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51556.38 MB 2025-02-15 12:51:38,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:51:38,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13113.49 MB 2025-02-15 12:51:38,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33890.06 MB 2025-02-15 12:51:38,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:51:38,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:51:38,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:51:38,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33890.06 MB 2025-02-15 12:51:38,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33990.53 MB 2025-02-15 12:51:38,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:51:38,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38442.89 MB 2025-02-15 12:51:38,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38442.89 MB 2025-02-15 12:51:38,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:51:38,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.33 MB 2025-02-15 12:51:38,844 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:51:38,844 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:51:38,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:51:38,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:51:38,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:51:38,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:51:38,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33990.53 MB 2025-02-15 12:51:38,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30521.73 MB 2025-02-15 12:51:38,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3468.80 MB 2025-02-15 12:51:38,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38442.89 MB 2025-02-15 12:51:38,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48932.85 MB 2025-02-15 12:51:38,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:51:38,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34716.03 MB 2025-02-15 12:51:39,009 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:51:39,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,010 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,011 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:51:39,016 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:51:39,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,017 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:51:39,017 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:51:39,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,017 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,018 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:39,024 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:51:39,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,024 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,025 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:39,025 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:51:39,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,025 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:39,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,026 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:51:39,026 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:51:39,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,026 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:51:39,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,029 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,030 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,031 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:51:39,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:51:39,036 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:05,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:05,513 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:05,518 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:52:05,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:05,519 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 626, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:52:05,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:05,520 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 626, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:52:15,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:52:15,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:52:15,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.63 seconds 2025-02-15 12:52:15,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:15,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28412.75 MB 2025-02-15 12:52:15,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30628.13 MB 2025-02-15 12:52:15,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2215.38 MB 2025-02-15 12:52:15,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53798.24 MB 2025-02-15 12:52:15,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35206.99 MB 2025-02-15 12:52:15,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18591.25 MB 2025-02-15 12:52:15,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39469.57 MB 2025-02-15 12:52:15,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:52:15,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:52:15,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 12:52:15,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:15,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30628.13 MB 2025-02-15 12:52:15,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30115.25 MB 2025-02-15 12:52:15,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -512.88 MB 2025-02-15 12:52:15,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35206.99 MB 2025-02-15 12:52:15,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41290.83 MB 2025-02-15 12:52:15,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6083.84 MB 2025-02-15 12:52:15,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38807.92 MB 2025-02-15 12:52:17,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:52:17,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:52:17,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:52:17,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30115.25 MB 2025-02-15 12:52:17,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30646.09 MB 2025-02-15 12:52:17,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:52:17,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41290.83 MB 2025-02-15 12:52:17,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37331.40 MB 2025-02-15 12:52:17,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3959.42 MB 2025-02-15 12:52:17,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34624.64 MB 2025-02-15 12:52:17,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:52:17,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:52:17,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:52:17,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30646.09 MB 2025-02-15 12:52:17,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32535.46 MB 2025-02-15 12:52:17,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.36 MB 2025-02-15 12:52:17,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37331.40 MB 2025-02-15 12:52:17,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37331.40 MB 2025-02-15 12:52:17,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:52:17,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33952.88 MB 2025-02-15 12:52:17,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:52:17,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:52:17,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:52:17,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32535.46 MB 2025-02-15 12:52:17,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34777.31 MB 2025-02-15 12:52:17,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:52:17,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37331.40 MB 2025-02-15 12:52:17,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42993.71 MB 2025-02-15 12:52:17,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:52:17,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40321.59 MB 2025-02-15 12:52:17,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:52:17,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:52:17,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:52:17,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30646.09 MB 2025-02-15 12:52:17,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34777.31 MB 2025-02-15 12:52:17,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.22 MB 2025-02-15 12:52:17,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37331.40 MB 2025-02-15 12:52:17,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42993.71 MB 2025-02-15 12:52:17,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:52:17,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40321.59 MB 2025-02-15 12:52:17,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:52:17,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:52:17,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:52:17,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35485.10 MB 2025-02-15 12:52:17,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36252.10 MB 2025-02-15 12:52:17,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:52:17,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42993.71 MB 2025-02-15 12:52:17,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-15 12:52:17,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:52:17,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36959.89 MB 2025-02-15 12:52:17,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:52:17,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:52:17,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:52:17,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36664.99 MB 2025-02-15 12:52:17,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36874.51 MB 2025-02-15 12:52:17,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.52 MB 2025-02-15 12:52:17,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43408.95 MB 2025-02-15 12:52:17,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-15 12:52:17,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:52:17,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37051.95 MB 2025-02-15 12:52:17,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:52:17,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:52:17,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.99 seconds 2025-02-15 12:52:17,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26231.71 MB 2025-02-15 12:52:17,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37075.58 MB 2025-02-15 12:52:17,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10843.87 MB 2025-02-15 12:52:17,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53798.24 MB 2025-02-15 12:52:17,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-15 12:52:17,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10389.29 MB 2025-02-15 12:52:17,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37075.58 MB 2025-02-15 12:52:17,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:52:17,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:52:17,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:52:17,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37075.58 MB 2025-02-15 12:52:17,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37176.05 MB 2025-02-15 12:52:17,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:52:17,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43408.95 MB 2025-02-15 12:52:17,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-15 12:52:17,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:52:17,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37778.85 MB 2025-02-15 12:52:17,794 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:52:17,794 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:52:17,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:52:17,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:52:17,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:52:17,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:52:17,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27494.64 MB 2025-02-15 12:52:17,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31689.12 MB 2025-02-15 12:52:17,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:52:17,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43408.95 MB 2025-02-15 12:52:17,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51799.65 MB 2025-02-15 12:52:17,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:52:17,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35883.42 MB 2025-02-15 12:52:17,960 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:52:17,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,961 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:52:17,967 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:52:17,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,969 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:52:17,969 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:52:17,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:17,977 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:52:17,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,978 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,978 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:17,978 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:52:17,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,979 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:17,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,980 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:52:17,980 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:52:17,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,980 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:52:17,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,984 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,985 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,987 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:52:17,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:52:17,992 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:04,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:04,535 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:04,540 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:53:04,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:04,541 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 679, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:53:04,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:04,542 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 679, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:53:14,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:53:14,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:53:14,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.42 seconds 2025-02-15 12:53:14,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:14,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28904.40 MB 2025-02-15 12:53:14,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31307.34 MB 2025-02-15 12:53:14,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2402.94 MB 2025-02-15 12:53:14,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56786.68 MB 2025-02-15 12:53:14,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35599.16 MB 2025-02-15 12:53:14,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21187.53 MB 2025-02-15 12:53:14,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40187.71 MB 2025-02-15 12:53:14,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:53:14,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:53:14,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:53:14,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:14,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31307.34 MB 2025-02-15 12:53:14,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28974.03 MB 2025-02-15 12:53:14,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2333.31 MB 2025-02-15 12:53:14,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35599.16 MB 2025-02-15 12:53:14,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-15 12:53:14,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 641.73 MB 2025-02-15 12:53:14,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33849.68 MB 2025-02-15 12:53:15,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:53:15,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:53:15,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-15 12:53:15,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:15,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28974.03 MB 2025-02-15 12:53:15,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29214.24 MB 2025-02-15 12:53:15,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.21 MB 2025-02-15 12:53:15,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36240.88 MB 2025-02-15 12:53:15,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34141.63 MB 2025-02-15 12:53:15,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2099.25 MB 2025-02-15 12:53:15,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33143.68 MB 2025-02-15 12:53:15,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:53:15,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:53:15,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:53:15,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:15,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29214.24 MB 2025-02-15 12:53:15,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30069.04 MB 2025-02-15 12:53:15,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 854.81 MB 2025-02-15 12:53:15,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34141.63 MB 2025-02-15 12:53:15,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34141.63 MB 2025-02-15 12:53:15,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:53:15,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30711.02 MB 2025-02-15 12:53:15,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:53:15,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:53:15,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:53:15,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:15,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30069.04 MB 2025-02-15 12:53:15,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31083.52 MB 2025-02-15 12:53:15,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.48 MB 2025-02-15 12:53:15,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34141.63 MB 2025-02-15 12:53:15,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34997.27 MB 2025-02-15 12:53:15,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 855.64 MB 2025-02-15 12:53:15,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33594.43 MB 2025-02-15 12:53:15,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:53:15,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:53:15,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 12:53:15,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:15,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29214.24 MB 2025-02-15 12:53:15,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31083.52 MB 2025-02-15 12:53:15,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1869.28 MB 2025-02-15 12:53:15,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34141.63 MB 2025-02-15 12:53:15,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34997.27 MB 2025-02-15 12:53:15,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 855.64 MB 2025-02-15 12:53:15,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33594.43 MB 2025-02-15 12:53:16,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:53:16,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:53:16,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:53:16,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:16,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31403.79 MB 2025-02-15 12:53:16,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31751.45 MB 2025-02-15 12:53:16,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.66 MB 2025-02-15 12:53:16,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34997.27 MB 2025-02-15 12:53:16,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35183.92 MB 2025-02-15 12:53:16,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-15 12:53:16,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32076.55 MB 2025-02-15 12:53:16,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:53:16,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:53:16,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:53:16,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:16,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31938.29 MB 2025-02-15 12:53:16,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32139.12 MB 2025-02-15 12:53:16,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 200.83 MB 2025-02-15 12:53:16,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35183.92 MB 2025-02-15 12:53:16,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35183.92 MB 2025-02-15 12:53:16,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:53:16,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32146.59 MB 2025-02-15 12:53:16,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:53:16,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:53:16,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.51 seconds 2025-02-15 12:53:16,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:16,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26538.71 MB 2025-02-15 12:53:16,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32339.78 MB 2025-02-15 12:53:16,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5801.07 MB 2025-02-15 12:53:16,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56786.68 MB 2025-02-15 12:53:16,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35183.92 MB 2025-02-15 12:53:16,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21602.76 MB 2025-02-15 12:53:16,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32339.78 MB 2025-02-15 12:53:16,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:53:16,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:53:16,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:53:16,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:16,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32339.78 MB 2025-02-15 12:53:16,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32440.03 MB 2025-02-15 12:53:16,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-15 12:53:16,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35183.92 MB 2025-02-15 12:53:16,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35183.92 MB 2025-02-15 12:53:16,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:53:16,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33041.58 MB 2025-02-15 12:53:16,334 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 12:53:16,334 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:53:16,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:53:16,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:53:16,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:53:16,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:53:16,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27220.40 MB 2025-02-15 12:53:16,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31406.32 MB 2025-02-15 12:53:16,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-15 12:53:16,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35183.92 MB 2025-02-15 12:53:16,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39369.83 MB 2025-02-15 12:53:16,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 12:53:16,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35591.57 MB 2025-02-15 12:53:16,502 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 12:53:16,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:53:16,509 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:53:16,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,510 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:53:16,510 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:53:16,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,511 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,511 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:16,517 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:53:16,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,518 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,518 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:16,518 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:53:16,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,519 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:16,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,519 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:53:16,519 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:53:16,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,520 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:16,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,523 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,524 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,525 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:53:16,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:16,529 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:55,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:55,952 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:53:55,957 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:53:55,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:55,959 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 938, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:53:55,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:53:55,959 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 938, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:54:10,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:54:10,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:54:10,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.52 seconds 2025-02-15 12:54:10,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:10,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30830.41 MB 2025-02-15 12:54:10,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34150.20 MB 2025-02-15 12:54:10,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3319.79 MB 2025-02-15 12:54:10,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44478.50 MB 2025-02-15 12:54:10,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40139.49 MB 2025-02-15 12:54:10,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4339.01 MB 2025-02-15 12:54:10,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43020.50 MB 2025-02-15 12:54:10,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:54:10,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:54:10,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:54:10,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:10,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34150.20 MB 2025-02-15 12:54:10,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31980.84 MB 2025-02-15 12:54:10,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2169.36 MB 2025-02-15 12:54:10,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40139.49 MB 2025-02-15 12:54:10,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-15 12:54:10,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8040.48 MB 2025-02-15 12:54:10,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44289.68 MB 2025-02-15 12:54:12,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:54:12,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:54:12,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:54:12,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31980.84 MB 2025-02-15 12:54:12,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32511.68 MB 2025-02-15 12:54:12,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:54:12,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48179.97 MB 2025-02-15 12:54:12,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38944.11 MB 2025-02-15 12:54:12,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9235.86 MB 2025-02-15 12:54:12,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36490.23 MB 2025-02-15 12:54:12,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:54:12,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:54:12,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:54:12,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32511.68 MB 2025-02-15 12:54:12,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34401.14 MB 2025-02-15 12:54:12,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.46 MB 2025-02-15 12:54:12,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38944.11 MB 2025-02-15 12:54:12,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39887.83 MB 2025-02-15 12:54:12,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:54:12,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35818.57 MB 2025-02-15 12:54:12,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:54:12,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:54:12,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:54:12,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34401.14 MB 2025-02-15 12:54:12,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36642.99 MB 2025-02-15 12:54:12,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:54:12,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39887.83 MB 2025-02-15 12:54:12,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45550.14 MB 2025-02-15 12:54:12,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:54:12,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42187.27 MB 2025-02-15 12:54:12,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:54:12,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:54:12,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:54:12,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32511.68 MB 2025-02-15 12:54:12,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36642.99 MB 2025-02-15 12:54:12,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.31 MB 2025-02-15 12:54:12,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38944.11 MB 2025-02-15 12:54:12,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45550.14 MB 2025-02-15 12:54:12,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 12:54:12,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42187.27 MB 2025-02-15 12:54:12,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:54:12,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:54:12,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:54:12,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37350.78 MB 2025-02-15 12:54:12,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38117.78 MB 2025-02-15 12:54:12,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:54:12,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45550.14 MB 2025-02-15 12:54:12,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45965.38 MB 2025-02-15 12:54:12,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:54:12,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38825.57 MB 2025-02-15 12:54:12,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:54:12,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:54:12,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:54:12,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38530.67 MB 2025-02-15 12:54:12,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38737.74 MB 2025-02-15 12:54:12,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.07 MB 2025-02-15 12:54:12,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45965.38 MB 2025-02-15 12:54:12,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45965.38 MB 2025-02-15 12:54:12,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:54:12,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38945.10 MB 2025-02-15 12:54:12,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:54:12,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:54:12,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.93 seconds 2025-02-15 12:54:12,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:12,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27562.34 MB 2025-02-15 12:54:12,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38938.82 MB 2025-02-15 12:54:12,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11376.48 MB 2025-02-15 12:54:12,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44478.50 MB 2025-02-15 12:54:12,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45965.38 MB 2025-02-15 12:54:12,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1486.88 MB 2025-02-15 12:54:12,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38945.10 MB 2025-02-15 12:54:13,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:54:13,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:54:13,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:54:13,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:13,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38938.82 MB 2025-02-15 12:54:13,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39039.28 MB 2025-02-15 12:54:13,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:54:13,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45965.38 MB 2025-02-15 12:54:13,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45965.38 MB 2025-02-15 12:54:13,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:54:13,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39642.08 MB 2025-02-15 12:54:13,175 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:54:13,176 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:54:13,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:54:13,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:54:13,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:54:13,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:13,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28825.26 MB 2025-02-15 12:54:13,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33019.75 MB 2025-02-15 12:54:13,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:54:13,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45965.38 MB 2025-02-15 12:54:13,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54356.08 MB 2025-02-15 12:54:13,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:54:13,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37214.05 MB 2025-02-15 12:54:13,347 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:54:13,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,349 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,350 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:54:13,354 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:54:13,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,355 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:54:13,356 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:54:13,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,356 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,357 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:13,363 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:54:13,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,364 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,364 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:13,364 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:54:13,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,365 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:13,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,365 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:54:13,365 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:54:13,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,366 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:13,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,369 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,370 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,371 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:13,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:13,376 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:36,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:36,467 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:36,472 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:54:36,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:36,474 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 823, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:54:36,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:36,474 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 823, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:54:49,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:54:49,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:54:49,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.79 seconds 2025-02-15 12:54:49,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:49,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30150.37 MB 2025-02-15 12:54:49,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33063.31 MB 2025-02-15 12:54:49,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2912.94 MB 2025-02-15 12:54:49,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59586.38 MB 2025-02-15 12:54:49,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36647.73 MB 2025-02-15 12:54:49,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22938.65 MB 2025-02-15 12:54:49,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41887.47 MB 2025-02-15 12:54:49,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:54:49,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:54:49,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:54:49,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:49,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33063.31 MB 2025-02-15 12:54:49,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30154.82 MB 2025-02-15 12:54:49,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2908.50 MB 2025-02-15 12:54:49,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36647.73 MB 2025-02-15 12:54:49,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37742.44 MB 2025-02-15 12:54:49,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1094.71 MB 2025-02-15 12:54:49,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35984.89 MB 2025-02-15 12:54:50,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:54:50,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:54:50,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-15 12:54:50,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30154.82 MB 2025-02-15 12:54:50,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30430.85 MB 2025-02-15 12:54:50,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-15 12:54:50,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37742.44 MB 2025-02-15 12:54:50,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34829.50 MB 2025-02-15 12:54:50,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2912.94 MB 2025-02-15 12:54:50,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34409.40 MB 2025-02-15 12:54:50,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:54:50,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:54:50,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:54:50,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30430.85 MB 2025-02-15 12:54:50,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31413.17 MB 2025-02-15 12:54:50,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-15 12:54:50,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34829.50 MB 2025-02-15 12:54:50,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34829.50 MB 2025-02-15 12:54:50,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:54:50,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.24 MB 2025-02-15 12:54:50,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:54:50,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:54:50,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 12:54:50,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31413.17 MB 2025-02-15 12:54:50,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32578.97 MB 2025-02-15 12:54:50,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1165.80 MB 2025-02-15 12:54:50,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34829.50 MB 2025-02-15 12:54:50,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36547.07 MB 2025-02-15 12:54:50,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-15 12:54:50,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35461.96 MB 2025-02-15 12:54:50,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:54:50,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:54:50,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 12:54:50,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30430.85 MB 2025-02-15 12:54:50,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32578.97 MB 2025-02-15 12:54:50,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.11 MB 2025-02-15 12:54:50,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34829.50 MB 2025-02-15 12:54:50,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36547.07 MB 2025-02-15 12:54:50,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-15 12:54:50,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35461.96 MB 2025-02-15 12:54:50,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:54:50,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:54:50,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 12:54:50,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32947.02 MB 2025-02-15 12:54:50,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33345.86 MB 2025-02-15 12:54:50,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-15 12:54:50,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36547.07 MB 2025-02-15 12:54:50,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-15 12:54:50,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 12:54:50,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33716.27 MB 2025-02-15 12:54:50,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:54:50,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:54:50,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:54:50,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33560.57 MB 2025-02-15 12:54:50,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33762.32 MB 2025-02-15 12:54:50,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.75 MB 2025-02-15 12:54:50,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36760.98 MB 2025-02-15 12:54:50,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-15 12:54:50,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:54:50,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33787.45 MB 2025-02-15 12:54:50,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:54:50,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:54:50,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.06 seconds 2025-02-15 12:54:50,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27282.97 MB 2025-02-15 12:54:50,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33962.97 MB 2025-02-15 12:54:50,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6680.00 MB 2025-02-15 12:54:50,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59586.38 MB 2025-02-15 12:54:50,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-15 12:54:50,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22825.40 MB 2025-02-15 12:54:50,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33962.97 MB 2025-02-15 12:54:50,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:54:50,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:54:50,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:54:50,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27935.50 MB 2025-02-15 12:54:50,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28035.75 MB 2025-02-15 12:54:50,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-15 12:54:50,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36760.98 MB 2025-02-15 12:54:50,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-15 12:54:50,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:54:50,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28637.77 MB 2025-02-15 12:54:50,819 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 12:54:50,819 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:54:50,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:54:50,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:54:50,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:54:50,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:54:50,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28035.75 MB 2025-02-15 12:54:50,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32221.67 MB 2025-02-15 12:54:50,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-15 12:54:50,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36760.98 MB 2025-02-15 12:54:50,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45132.81 MB 2025-02-15 12:54:50,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 12:54:50,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36407.59 MB 2025-02-15 12:54:50,990 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 12:54:50,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:50,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:50,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:50,992 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:54:50,997 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:54:50,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:50,998 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:54:50,998 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:54:50,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:50,999 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:50,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:50,999 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:51,005 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:54:51,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,006 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:51,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,006 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:51,006 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:54:51,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,007 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:51,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,007 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:54:51,007 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:54:51,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,008 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:54:51,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,013 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:51,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,014 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:51,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,015 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:54:51,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:54:51,021 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:07,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:07,861 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:07,866 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:55:07,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:07,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 538, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:55:07,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:07,868 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 538, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:55:16,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:55:16,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:55:16,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.34 seconds 2025-02-15 12:55:16,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28286.69 MB 2025-02-15 12:55:16,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30190.91 MB 2025-02-15 12:55:16,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1904.21 MB 2025-02-15 12:55:16,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50482.64 MB 2025-02-15 12:55:16,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35758.54 MB 2025-02-15 12:55:16,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14724.10 MB 2025-02-15 12:55:16,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39117.02 MB 2025-02-15 12:55:16,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:55:16,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:55:16,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:55:16,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30190.91 MB 2025-02-15 12:55:16,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28247.65 MB 2025-02-15 12:55:16,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1943.26 MB 2025-02-15 12:55:16,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35758.54 MB 2025-02-15 12:55:16,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35758.54 MB 2025-02-15 12:55:16,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:55:16,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32016.89 MB 2025-02-15 12:55:16,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:55:16,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:55:16,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.65 seconds 2025-02-15 12:55:16,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28247.65 MB 2025-02-15 12:55:16,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28420.17 MB 2025-02-15 12:55:16,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-15 12:55:16,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35758.54 MB 2025-02-15 12:55:16,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32438.75 MB 2025-02-15 12:55:16,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3319.79 MB 2025-02-15 12:55:16,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32418.33 MB 2025-02-15 12:55:16,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:55:16,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:55:16,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:55:16,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28420.17 MB 2025-02-15 12:55:16,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29034.12 MB 2025-02-15 12:55:16,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-15 12:55:16,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32438.75 MB 2025-02-15 12:55:16,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32438.75 MB 2025-02-15 12:55:16,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:55:16,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29494.79 MB 2025-02-15 12:55:16,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:55:16,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:55:16,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 12:55:16,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29034.12 MB 2025-02-15 12:55:16,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29762.77 MB 2025-02-15 12:55:16,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-15 12:55:16,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32438.75 MB 2025-02-15 12:55:16,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32747.03 MB 2025-02-15 12:55:16,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 308.28 MB 2025-02-15 12:55:16,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31564.61 MB 2025-02-15 12:55:16,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:55:16,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:55:16,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:55:16,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:16,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28420.17 MB 2025-02-15 12:55:16,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29762.77 MB 2025-02-15 12:55:16,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-15 12:55:16,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32438.75 MB 2025-02-15 12:55:16,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32747.03 MB 2025-02-15 12:55:16,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 308.28 MB 2025-02-15 12:55:16,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31564.61 MB 2025-02-15 12:55:17,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:55:17,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:55:17,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:55:17,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:17,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29992.80 MB 2025-02-15 12:55:17,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30242.07 MB 2025-02-15 12:55:17,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-15 12:55:17,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32747.03 MB 2025-02-15 12:55:17,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32879.15 MB 2025-02-15 12:55:17,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-15 12:55:17,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.64 MB 2025-02-15 12:55:17,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:55:17,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:55:17,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:55:17,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:17,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30376.27 MB 2025-02-15 12:55:17,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30553.51 MB 2025-02-15 12:55:17,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.24 MB 2025-02-15 12:55:17,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32879.15 MB 2025-02-15 12:55:17,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32883.34 MB 2025-02-15 12:55:17,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 12:55:17,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30553.51 MB 2025-02-15 12:55:17,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:55:17,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:55:17,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.16 seconds 2025-02-15 12:55:17,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:17,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26412.26 MB 2025-02-15 12:55:17,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30729.60 MB 2025-02-15 12:55:17,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4317.34 MB 2025-02-15 12:55:17,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50482.64 MB 2025-02-15 12:55:17,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32883.34 MB 2025-02-15 12:55:17,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17599.30 MB 2025-02-15 12:55:17,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30729.60 MB 2025-02-15 12:55:17,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:55:17,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:55:17,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:55:17,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:17,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26845.42 MB 2025-02-15 12:55:17,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26933.40 MB 2025-02-15 12:55:17,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 87.98 MB 2025-02-15 12:55:17,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32883.34 MB 2025-02-15 12:55:17,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32883.34 MB 2025-02-15 12:55:17,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:55:17,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27461.29 MB 2025-02-15 12:55:17,268 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7146, cut from 7148 2025-02-15 12:55:17,268 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:55:17,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:55:17,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:55:17,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:55:17,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:55:17,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26933.40 MB 2025-02-15 12:55:17,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30607.61 MB 2025-02-15 12:55:17,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3674.21 MB 2025-02-15 12:55:17,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32883.34 MB 2025-02-15 12:55:17,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42068.87 MB 2025-02-15 12:55:17,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9185.53 MB 2025-02-15 12:55:17,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34280.35 MB 2025-02-15 12:55:17,412 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6938] 2025-02-15 12:55:17,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,413 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:55:17,419 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:55:17,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,422 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:55:17,422 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 12:55:17,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,423 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,424 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:17,429 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:55:17,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,430 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,430 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:17,430 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:55:17,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,431 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:17,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,431 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:55:17,431 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:55:17,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,432 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:55:17,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,435 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,436 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,437 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:55:17,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:55:17,442 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:12,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:12,685 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:12,690 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:56:12,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:12,691 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:56:12,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:12,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:56:18,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:56:18,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:56:18,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.97 seconds 2025-02-15 12:56:18,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:18,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27348.92 MB 2025-02-15 12:56:18,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28714.95 MB 2025-02-15 12:56:18,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1366.03 MB 2025-02-15 12:56:18,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47540.34 MB 2025-02-15 12:56:18,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:18,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13342.08 MB 2025-02-15 12:56:18,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37726.26 MB 2025-02-15 12:56:18,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:56:18,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:56:18,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:56:18,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:18,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28714.95 MB 2025-02-15 12:56:18,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27066.22 MB 2025-02-15 12:56:18,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1648.74 MB 2025-02-15 12:56:18,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:18,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:18,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:18,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29515.64 MB 2025-02-15 12:56:18,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:56:18,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:56:18,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:56:18,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:18,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27066.22 MB 2025-02-15 12:56:18,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27141.86 MB 2025-02-15 12:56:18,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 75.64 MB 2025-02-15 12:56:18,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:18,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:18,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:18,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30704.17 MB 2025-02-15 12:56:18,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:56:18,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:56:18,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:56:18,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:18,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27141.80 MB 2025-02-15 12:56:18,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27410.99 MB 2025-02-15 12:56:18,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.19 MB 2025-02-15 12:56:18,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:18,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:18,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:18,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27612.98 MB 2025-02-15 12:56:19,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:56:19,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:56:19,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 12:56:19,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27410.99 MB 2025-02-15 12:56:19,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27739.08 MB 2025-02-15 12:56:19,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 328.09 MB 2025-02-15 12:56:19,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:19,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:19,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:19,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28521.61 MB 2025-02-15 12:56:19,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:56:19,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:56:19,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:56:19,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27141.80 MB 2025-02-15 12:56:19,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27739.08 MB 2025-02-15 12:56:19,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 597.28 MB 2025-02-15 12:56:19,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:19,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34198.26 MB 2025-02-15 12:56:19,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:19,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28521.61 MB 2025-02-15 12:56:19,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:56:19,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:56:19,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:56:19,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27885.75 MB 2025-02-15 12:56:19,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28023.07 MB 2025-02-15 12:56:19,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 137.32 MB 2025-02-15 12:56:19,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34198.26 MB 2025-02-15 12:56:19,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 12:56:19,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 83.89 MB 2025-02-15 12:56:19,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28123.93 MB 2025-02-15 12:56:19,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:56:19,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:56:19,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 12:56:19,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28110.01 MB 2025-02-15 12:56:19,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28232.67 MB 2025-02-15 12:56:19,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 122.67 MB 2025-02-15 12:56:19,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-15 12:56:19,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 12:56:19,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:19,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28232.67 MB 2025-02-15 12:56:19,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:56:19,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:56:19,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.37 seconds 2025-02-15 12:56:19,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26004.07 MB 2025-02-15 12:56:19,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28356.25 MB 2025-02-15 12:56:19,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2352.18 MB 2025-02-15 12:56:19,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47540.34 MB 2025-02-15 12:56:19,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 12:56:19,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13258.19 MB 2025-02-15 12:56:19,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28356.25 MB 2025-02-15 12:56:19,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:56:19,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:56:19,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:56:19,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28356.25 MB 2025-02-15 12:56:19,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28418.00 MB 2025-02-15 12:56:19,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 61.75 MB 2025-02-15 12:56:19,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-15 12:56:19,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-15 12:56:19,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:19,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28788.48 MB 2025-02-15 12:56:19,225 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5011, cut from 5013 2025-02-15 12:56:19,225 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for the video is 2 ('] 2025-02-15 12:56:19,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:56:19,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:56:19,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:56:19,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:19,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28418.00 MB 2025-02-15 12:56:19,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30995.94 MB 2025-02-15 12:56:19,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2577.95 MB 2025-02-15 12:56:19,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-15 12:56:19,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36861.64 MB 2025-02-15 12:56:19,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2579.50 MB 2025-02-15 12:56:19,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33573.38 MB 2025-02-15 12:56:19,329 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 4803] 2025-02-15 12:56:19,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,330 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,331 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:56:19,336 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:56:19,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,337 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:56:19,337 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for the video is 2 ('] 2025-02-15 12:56:19,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,338 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,338 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:19,344 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:56:19,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,345 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,345 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:19,345 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:56:19,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,346 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:19,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,346 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:56:19,346 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:56:19,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,347 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:19,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,352 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,353 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,354 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:19,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:19,360 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:39,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:39,871 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:39,876 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:56:39,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:39,877 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1014, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:56:39,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:39,878 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1014, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:56:55,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:56:55,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:56:55,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.68 seconds 2025-02-15 12:56:55,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:55,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31846.82 MB 2025-02-15 12:56:55,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35435.31 MB 2025-02-15 12:56:55,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3588.49 MB 2025-02-15 12:56:55,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42454.75 MB 2025-02-15 12:56:55,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40137.39 MB 2025-02-15 12:56:55,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2317.35 MB 2025-02-15 12:56:55,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44263.40 MB 2025-02-15 12:56:55,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:56:55,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:56:55,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 12:56:55,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:55,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35435.31 MB 2025-02-15 12:56:55,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32861.72 MB 2025-02-15 12:56:55,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2573.59 MB 2025-02-15 12:56:55,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40137.39 MB 2025-02-15 12:56:55,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48341.45 MB 2025-02-15 12:56:55,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8204.06 MB 2025-02-15 12:56:55,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46444.78 MB 2025-02-15 12:56:57,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:56:57,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:56:57,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 12:56:57,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32861.72 MB 2025-02-15 12:56:57,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33392.57 MB 2025-02-15 12:56:57,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:56:57,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48341.45 MB 2025-02-15 12:56:57,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37962.65 MB 2025-02-15 12:56:57,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10378.81 MB 2025-02-15 12:56:57,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37371.11 MB 2025-02-15 12:56:57,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:56:57,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:56:57,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:56:57,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33392.57 MB 2025-02-15 12:56:57,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35282.06 MB 2025-02-15 12:56:57,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 12:56:57,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37962.65 MB 2025-02-15 12:56:57,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39850.08 MB 2025-02-15 12:56:57,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:56:57,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36699.49 MB 2025-02-15 12:56:57,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:56:57,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:56:57,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:56:57,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35282.06 MB 2025-02-15 12:56:57,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37523.92 MB 2025-02-15 12:56:57,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:56:57,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39850.08 MB 2025-02-15 12:56:57,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45984.25 MB 2025-02-15 12:56:57,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:56:57,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43068.20 MB 2025-02-15 12:56:57,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:56:57,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:56:57,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 12:56:57,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33392.57 MB 2025-02-15 12:56:57,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37523.92 MB 2025-02-15 12:56:57,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 12:56:57,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37962.65 MB 2025-02-15 12:56:57,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45984.25 MB 2025-02-15 12:56:57,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 12:56:57,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43068.20 MB 2025-02-15 12:56:57,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:56:57,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:56:57,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 12:56:57,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38231.70 MB 2025-02-15 12:56:57,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38998.71 MB 2025-02-15 12:56:57,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:56:57,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45984.25 MB 2025-02-15 12:56:57,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46399.49 MB 2025-02-15 12:56:57,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:56:57,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39706.49 MB 2025-02-15 12:56:57,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:56:57,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:56:57,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:56:57,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39411.59 MB 2025-02-15 12:56:57,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39617.75 MB 2025-02-15 12:56:57,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.15 MB 2025-02-15 12:56:57,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46399.49 MB 2025-02-15 12:56:57,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46399.49 MB 2025-02-15 12:56:57,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:57,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39815.64 MB 2025-02-15 12:56:57,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:56:57,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:56:57,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.09 seconds 2025-02-15 12:56:57,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:57,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28313.97 MB 2025-02-15 12:56:57,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39818.60 MB 2025-02-15 12:56:57,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11504.63 MB 2025-02-15 12:56:57,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42454.75 MB 2025-02-15 12:56:57,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46399.49 MB 2025-02-15 12:56:57,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3944.74 MB 2025-02-15 12:56:57,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39818.60 MB 2025-02-15 12:56:58,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:56:58,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:56:58,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:56:58,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:58,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39818.60 MB 2025-02-15 12:56:58,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39918.96 MB 2025-02-15 12:56:58,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.36 MB 2025-02-15 12:56:58,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46399.49 MB 2025-02-15 12:56:58,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46399.49 MB 2025-02-15 12:56:58,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:56:58,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40521.09 MB 2025-02-15 12:56:58,255 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-15 12:56:58,255 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:56:58,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:56:58,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:56:58,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:56:58,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:56:58,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29576.67 MB 2025-02-15 12:56:58,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33766.78 MB 2025-02-15 12:56:58,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.11 MB 2025-02-15 12:56:58,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46399.49 MB 2025-02-15 12:56:58,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56874.76 MB 2025-02-15 12:56:58,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 12:56:58,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37956.89 MB 2025-02-15 12:56:58,420 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-15 12:56:58,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,421 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,422 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:56:58,429 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:56:58,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,430 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:56:58,430 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:56:58,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,431 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,431 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:58,437 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:56:58,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,438 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,438 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:58,438 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:56:58,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,439 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:58,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,439 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:56:58,439 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:56:58,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,440 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:56:58,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,443 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,444 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,446 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:56:58,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:56:58,452 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:33,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:33,194 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:33,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:57:33,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:33,201 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 452, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:57:33,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:33,202 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 452, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:57:40,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:57:40,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:57:40,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.02 seconds 2025-02-15 12:57:40,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:40,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28051.99 MB 2025-02-15 12:57:40,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29651.59 MB 2025-02-15 12:57:40,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1599.60 MB 2025-02-15 12:57:40,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62589.50 MB 2025-02-15 12:57:40,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34028.39 MB 2025-02-15 12:57:40,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28561.11 MB 2025-02-15 12:57:40,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38656.21 MB 2025-02-15 12:57:40,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:57:40,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:57:40,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 12:57:40,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:40,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29651.59 MB 2025-02-15 12:57:40,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30062.38 MB 2025-02-15 12:57:40,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.79 MB 2025-02-15 12:57:40,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34028.39 MB 2025-02-15 12:57:40,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39185.29 MB 2025-02-15 12:57:40,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5156.90 MB 2025-02-15 12:57:40,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36981.86 MB 2025-02-15 12:57:42,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:57:42,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:57:42,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:57:42,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30062.38 MB 2025-02-15 12:57:42,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30593.22 MB 2025-02-15 12:57:42,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:57:42,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39185.29 MB 2025-02-15 12:57:42,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35917.92 MB 2025-02-15 12:57:42,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3267.36 MB 2025-02-15 12:57:42,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34571.77 MB 2025-02-15 12:57:42,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:57:42,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:57:42,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:57:42,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30593.22 MB 2025-02-15 12:57:42,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32482.71 MB 2025-02-15 12:57:42,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 12:57:42,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35917.92 MB 2025-02-15 12:57:42,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36861.64 MB 2025-02-15 12:57:42,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 12:57:42,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33900.14 MB 2025-02-15 12:57:42,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:57:42,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:57:42,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:57:42,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32482.71 MB 2025-02-15 12:57:42,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34724.57 MB 2025-02-15 12:57:42,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:57:42,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36861.64 MB 2025-02-15 12:57:42,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42995.81 MB 2025-02-15 12:57:42,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 12:57:42,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40268.85 MB 2025-02-15 12:57:42,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:57:42,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:57:42,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:57:42,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30593.22 MB 2025-02-15 12:57:42,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34724.57 MB 2025-02-15 12:57:42,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 12:57:42,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35917.92 MB 2025-02-15 12:57:42,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42995.81 MB 2025-02-15 12:57:42,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 12:57:42,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40268.85 MB 2025-02-15 12:57:42,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:57:42,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:57:42,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 12:57:42,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35432.36 MB 2025-02-15 12:57:42,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36199.36 MB 2025-02-15 12:57:42,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:57:42,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42995.81 MB 2025-02-15 12:57:42,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43411.05 MB 2025-02-15 12:57:42,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:57:42,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36907.15 MB 2025-02-15 12:57:42,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:57:42,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:57:42,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:57:42,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36612.25 MB 2025-02-15 12:57:42,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36819.71 MB 2025-02-15 12:57:42,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.46 MB 2025-02-15 12:57:42,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43411.05 MB 2025-02-15 12:57:42,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43411.05 MB 2025-02-15 12:57:42,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:57:42,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37006.64 MB 2025-02-15 12:57:42,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:57:42,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:57:42,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.39 seconds 2025-02-15 12:57:42,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26477.18 MB 2025-02-15 12:57:42,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37020.78 MB 2025-02-15 12:57:42,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10543.60 MB 2025-02-15 12:57:42,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62589.50 MB 2025-02-15 12:57:42,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43411.05 MB 2025-02-15 12:57:42,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19178.46 MB 2025-02-15 12:57:42,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37020.78 MB 2025-02-15 12:57:42,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:57:42,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:57:42,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 12:57:42,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37020.78 MB 2025-02-15 12:57:42,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37121.24 MB 2025-02-15 12:57:42,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:57:42,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43411.05 MB 2025-02-15 12:57:42,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43411.05 MB 2025-02-15 12:57:42,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:57:42,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37724.04 MB 2025-02-15 12:57:42,888 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:57:42,888 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:57:42,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:57:42,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:57:42,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:57:42,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:57:42,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27740.10 MB 2025-02-15 12:57:42,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31934.59 MB 2025-02-15 12:57:42,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:57:42,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43411.05 MB 2025-02-15 12:57:42,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53901.00 MB 2025-02-15 12:57:42,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 12:57:42,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36128.89 MB 2025-02-15 12:57:43,147 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:57:43,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,151 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:57:43,159 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:57:43,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,161 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:57:43,161 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:57:43,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,162 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,164 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:43,173 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:57:43,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,174 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,175 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:43,175 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:57:43,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,176 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:43,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,177 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:57:43,177 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:57:43,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,178 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:57:43,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,189 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,192 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,195 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:57:43,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:57:43,202 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:40,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:40,667 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:40,672 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:58:40,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:40,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 769, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:58:40,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:40,674 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 769, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 12:58:52,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 12:58:52,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 12:58:52,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.92 seconds 2025-02-15 12:58:52,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:52,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30383.03 MB 2025-02-15 12:58:52,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33105.13 MB 2025-02-15 12:58:52,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2722.10 MB 2025-02-15 12:58:52,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59737.37 MB 2025-02-15 12:58:52,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37555.80 MB 2025-02-15 12:58:52,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22181.58 MB 2025-02-15 12:58:52,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42120.13 MB 2025-02-15 12:58:52,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 12:58:52,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 12:58:52,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 12:58:52,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:52,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33105.13 MB 2025-02-15 12:58:52,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31832.50 MB 2025-02-15 12:58:52,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1272.63 MB 2025-02-15 12:58:52,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37555.80 MB 2025-02-15 12:58:52,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44704.99 MB 2025-02-15 12:58:52,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7149.19 MB 2025-02-15 12:58:52,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42354.24 MB 2025-02-15 12:58:54,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 12:58:54,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 12:58:54,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 12:58:54,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:54,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31832.50 MB 2025-02-15 12:58:54,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32363.34 MB 2025-02-15 12:58:54,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 12:58:54,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44704.99 MB 2025-02-15 12:58:54,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36958.11 MB 2025-02-15 12:58:54,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7746.88 MB 2025-02-15 12:58:54,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36341.89 MB 2025-02-15 12:58:54,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 12:58:54,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 12:58:54,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 12:58:54,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:54,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32363.34 MB 2025-02-15 12:58:54,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34252.83 MB 2025-02-15 12:58:54,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 12:58:54,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36958.11 MB 2025-02-15 12:58:54,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-15 12:58:54,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 12:58:54,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35670.26 MB 2025-02-15 12:58:54,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 12:58:54,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 12:58:54,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 12:58:54,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:54,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34252.83 MB 2025-02-15 12:58:54,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36494.69 MB 2025-02-15 12:58:54,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 12:58:54,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-15 12:58:54,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44507.86 MB 2025-02-15 12:58:54,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 12:58:54,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42038.97 MB 2025-02-15 12:58:54,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 12:58:54,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 12:58:54,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 12:58:54,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:54,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32363.34 MB 2025-02-15 12:58:54,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36494.69 MB 2025-02-15 12:58:54,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 12:58:54,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36958.11 MB 2025-02-15 12:58:54,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44507.86 MB 2025-02-15 12:58:54,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 12:58:54,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42038.97 MB 2025-02-15 12:58:54,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 12:58:54,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 12:58:54,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 12:58:54,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:54,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37202.48 MB 2025-02-15 12:58:54,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37969.48 MB 2025-02-15 12:58:54,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 12:58:54,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44507.86 MB 2025-02-15 12:58:54,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44923.09 MB 2025-02-15 12:58:54,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 12:58:54,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38677.27 MB 2025-02-15 12:58:55,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 12:58:55,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 12:58:55,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:58:55,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:55,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38382.37 MB 2025-02-15 12:58:55,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38590.31 MB 2025-02-15 12:58:55,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.94 MB 2025-02-15 12:58:55,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44923.09 MB 2025-02-15 12:58:55,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44923.09 MB 2025-02-15 12:58:55,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:58:55,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38785.67 MB 2025-02-15 12:58:55,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 12:58:55,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 12:58:55,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.35 seconds 2025-02-15 12:58:55,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:55,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27703.77 MB 2025-02-15 12:58:55,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38791.38 MB 2025-02-15 12:58:55,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11087.61 MB 2025-02-15 12:58:55,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59737.37 MB 2025-02-15 12:58:55,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44923.09 MB 2025-02-15 12:58:55,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14814.28 MB 2025-02-15 12:58:55,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38791.38 MB 2025-02-15 12:58:55,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 12:58:55,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 12:58:55,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 12:58:55,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:55,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38791.38 MB 2025-02-15 12:58:55,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38891.85 MB 2025-02-15 12:58:55,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 12:58:55,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44923.09 MB 2025-02-15 12:58:55,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44923.09 MB 2025-02-15 12:58:55,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 12:58:55,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39494.65 MB 2025-02-15 12:58:55,304 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 12:58:55,305 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:58:55,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 12:58:55,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 12:58:55,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 12:58:55,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 12:58:55,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28966.69 MB 2025-02-15 12:58:55,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33161.18 MB 2025-02-15 12:58:55,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 12:58:55,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44923.09 MB 2025-02-15 12:58:55,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53313.80 MB 2025-02-15 12:58:55,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 12:58:55,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37355.48 MB 2025-02-15 12:58:55,471 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 12:58:55,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,473 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,473 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 12:58:55,478 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 12:58:55,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,479 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 12:58:55,479 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 12:58:55,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,480 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,481 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:55,486 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 12:58:55,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,487 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,487 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:55,487 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 12:58:55,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,488 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:55,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,488 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 12:58:55,489 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 12:58:55,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,489 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:58:55,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,492 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,493 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,494 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 12:58:55,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:58:55,500 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:59:43,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:59:43,992 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 12:59:43,997 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 12:59:43,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:59:43,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 12:59:43,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 12:59:43,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:00:04,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:00:04,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:00:04,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.03 seconds 2025-02-15 13:00:04,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:04,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34211.49 MB 2025-02-15 13:00:04,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38815.66 MB 2025-02-15 13:00:04,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4604.17 MB 2025-02-15 13:00:04,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59271.81 MB 2025-02-15 13:00:04,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46076.53 MB 2025-02-15 13:00:04,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13195.28 MB 2025-02-15 13:00:04,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47759.72 MB 2025-02-15 13:00:04,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:00:04,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:00:04,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:00:04,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:04,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38815.66 MB 2025-02-15 13:00:04,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34718.55 MB 2025-02-15 13:00:04,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4097.10 MB 2025-02-15 13:00:04,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46076.53 MB 2025-02-15 13:00:04,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56470.01 MB 2025-02-15 13:00:04,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10393.49 MB 2025-02-15 13:00:04,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52314.93 MB 2025-02-15 13:00:06,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:00:06,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:00:06,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:00:06,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34718.55 MB 2025-02-15 13:00:06,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35249.39 MB 2025-02-15 13:00:06,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:00:06,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56470.01 MB 2025-02-15 13:00:06,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40307.26 MB 2025-02-15 13:00:06,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16162.75 MB 2025-02-15 13:00:06,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39227.94 MB 2025-02-15 13:00:06,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:00:06,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:00:06,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:00:06,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35249.39 MB 2025-02-15 13:00:06,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37138.89 MB 2025-02-15 13:00:06,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:00:06,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40307.26 MB 2025-02-15 13:00:06,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41250.98 MB 2025-02-15 13:00:06,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 13:00:06,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38556.31 MB 2025-02-15 13:00:06,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:00:06,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:00:06,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 13:00:06,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37138.89 MB 2025-02-15 13:00:06,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39380.74 MB 2025-02-15 13:00:06,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:00:06,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41250.98 MB 2025-02-15 13:00:06,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47385.15 MB 2025-02-15 13:00:06,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:00:06,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44925.02 MB 2025-02-15 13:00:06,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:00:06,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:00:06,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:00:06,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35249.39 MB 2025-02-15 13:00:06,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39380.74 MB 2025-02-15 13:00:06,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:00:06,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40307.26 MB 2025-02-15 13:00:06,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47385.15 MB 2025-02-15 13:00:06,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 13:00:06,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44925.02 MB 2025-02-15 13:00:06,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:00:06,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:00:06,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:00:06,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40088.53 MB 2025-02-15 13:00:06,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40855.53 MB 2025-02-15 13:00:06,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:00:06,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47385.15 MB 2025-02-15 13:00:06,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-15 13:00:06,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:00:06,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41563.32 MB 2025-02-15 13:00:06,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:00:06,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:00:06,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:00:06,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41268.42 MB 2025-02-15 13:00:06,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41473.92 MB 2025-02-15 13:00:06,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.50 MB 2025-02-15 13:00:06,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-15 13:00:06,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-15 13:00:06,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:00:06,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41693.66 MB 2025-02-15 13:00:06,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:00:06,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:00:06,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.45 seconds 2025-02-15 13:00:06,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29678.70 MB 2025-02-15 13:00:06,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41673.96 MB 2025-02-15 13:00:06,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11995.26 MB 2025-02-15 13:00:06,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59271.81 MB 2025-02-15 13:00:06,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-15 13:00:06,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11471.42 MB 2025-02-15 13:00:06,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41693.66 MB 2025-02-15 13:00:06,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:00:06,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:00:06,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:00:06,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41673.96 MB 2025-02-15 13:00:06,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41773.92 MB 2025-02-15 13:00:06,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.95 MB 2025-02-15 13:00:06,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-15 13:00:06,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-15 13:00:06,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:00:06,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42373.62 MB 2025-02-15 13:00:06,730 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 13:00:06,731 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:00:06,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:00:06,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:00:06,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:00:06,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:00:06,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30940.59 MB 2025-02-15 13:00:06,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35113.92 MB 2025-02-15 13:00:06,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.33 MB 2025-02-15 13:00:06,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-15 13:00:06,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56147.05 MB 2025-02-15 13:00:06,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 13:00:06,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39287.26 MB 2025-02-15 13:00:06,897 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 13:00:06,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,898 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,899 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:00:06,904 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:00:06,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,905 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:00:06,905 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:00:06,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,906 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,906 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:00:06,912 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:00:06,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,912 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,913 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:00:06,913 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:00:06,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,913 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:00:06,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,914 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:00:06,914 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:00:06,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,915 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:00:06,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,918 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,919 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,919 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:00:06,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:00:06,925 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:00,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:00,731 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:00,736 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:01:00,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:00,738 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:01:00,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:00,738 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:01:19,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:01:19,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:01:19,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.71 seconds 2025-02-15 13:01:19,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:19,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33727.21 MB 2025-02-15 13:01:19,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38024.28 MB 2025-02-15 13:01:19,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4297.06 MB 2025-02-15 13:01:19,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62226.69 MB 2025-02-15 13:01:19,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43289.41 MB 2025-02-15 13:01:19,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18937.28 MB 2025-02-15 13:01:19,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46823.27 MB 2025-02-15 13:01:19,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:01:19,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:01:19,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 13:01:19,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:19,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38024.28 MB 2025-02-15 13:01:19,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34389.27 MB 2025-02-15 13:01:19,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3635.01 MB 2025-02-15 13:01:19,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43289.41 MB 2025-02-15 13:01:19,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53942.94 MB 2025-02-15 13:01:19,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10653.53 MB 2025-02-15 13:01:19,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50849.06 MB 2025-02-15 13:01:21,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:01:21,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:01:21,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:01:21,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34389.27 MB 2025-02-15 13:01:21,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34920.11 MB 2025-02-15 13:01:21,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:01:21,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53942.94 MB 2025-02-15 13:01:21,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41116.76 MB 2025-02-15 13:01:21,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12826.18 MB 2025-02-15 13:01:21,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38898.66 MB 2025-02-15 13:01:21,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:01:21,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:01:21,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:01:21,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34920.11 MB 2025-02-15 13:01:21,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36809.60 MB 2025-02-15 13:01:21,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:01:21,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41116.76 MB 2025-02-15 13:01:21,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42060.48 MB 2025-02-15 13:01:21,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 13:01:21,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38227.03 MB 2025-02-15 13:01:21,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:01:21,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:01:21,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:01:21,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36809.60 MB 2025-02-15 13:01:21,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39051.46 MB 2025-02-15 13:01:21,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:01:21,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42060.48 MB 2025-02-15 13:01:21,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47250.93 MB 2025-02-15 13:01:21,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:01:21,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44595.74 MB 2025-02-15 13:01:21,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:01:21,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:01:21,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:01:21,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34920.11 MB 2025-02-15 13:01:21,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39051.46 MB 2025-02-15 13:01:21,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:01:21,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41116.76 MB 2025-02-15 13:01:21,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47250.93 MB 2025-02-15 13:01:21,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:01:21,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44595.74 MB 2025-02-15 13:01:21,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:01:21,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:01:21,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:01:21,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39759.25 MB 2025-02-15 13:01:21,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40526.25 MB 2025-02-15 13:01:21,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:01:21,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47250.93 MB 2025-02-15 13:01:21,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47666.17 MB 2025-02-15 13:01:21,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:01:21,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41234.04 MB 2025-02-15 13:01:21,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:01:21,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:01:21,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:01:21,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40939.14 MB 2025-02-15 13:01:21,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41145.77 MB 2025-02-15 13:01:21,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.63 MB 2025-02-15 13:01:21,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47666.17 MB 2025-02-15 13:01:21,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47666.17 MB 2025-02-15 13:01:21,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:01:21,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41366.66 MB 2025-02-15 13:01:21,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:01:21,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:01:21,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.16 seconds 2025-02-15 13:01:21,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:21,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29497.54 MB 2025-02-15 13:01:21,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41346.52 MB 2025-02-15 13:01:21,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11848.98 MB 2025-02-15 13:01:21,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62226.69 MB 2025-02-15 13:01:21,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47666.17 MB 2025-02-15 13:01:21,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14560.53 MB 2025-02-15 13:01:21,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41366.66 MB 2025-02-15 13:01:22,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:01:22,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:01:22,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:01:22,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:22,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41346.52 MB 2025-02-15 13:01:22,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41446.83 MB 2025-02-15 13:01:22,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.31 MB 2025-02-15 13:01:22,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47666.17 MB 2025-02-15 13:01:22,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47666.17 MB 2025-02-15 13:01:22,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:01:22,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42048.67 MB 2025-02-15 13:01:22,180 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 13:01:22,180 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:01:22,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:01:22,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:01:22,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:01:22,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:01:22,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30760.14 MB 2025-02-15 13:01:22,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34948.15 MB 2025-02-15 13:01:22,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.01 MB 2025-02-15 13:01:22,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47666.17 MB 2025-02-15 13:01:22,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56042.19 MB 2025-02-15 13:01:22,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 13:01:22,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39136.17 MB 2025-02-15 13:01:22,348 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 13:01:22,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,349 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,350 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:01:22,355 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:01:22,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,356 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:01:22,356 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:01:22,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,356 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,357 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:22,363 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:01:22,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,363 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,364 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:22,364 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:01:22,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,364 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:22,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,365 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:01:22,365 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:01:22,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,365 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:22,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,372 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,374 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,376 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:01:22,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:22,382 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:58,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:58,449 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:01:58,454 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:01:58,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:58,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1178, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:01:58,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:01:58,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1178, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:02:16,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:02:16,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:02:16,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.25 seconds 2025-02-15 13:02:16,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:16,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33598.07 MB 2025-02-15 13:02:16,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37767.21 MB 2025-02-15 13:02:16,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4169.14 MB 2025-02-15 13:02:16,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62243.47 MB 2025-02-15 13:02:16,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43283.12 MB 2025-02-15 13:02:16,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18960.35 MB 2025-02-15 13:02:16,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46693.32 MB 2025-02-15 13:02:16,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:02:16,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:02:16,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 13:02:16,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:16,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37767.21 MB 2025-02-15 13:02:16,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34323.82 MB 2025-02-15 13:02:16,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3443.38 MB 2025-02-15 13:02:16,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43283.12 MB 2025-02-15 13:02:16,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53682.90 MB 2025-02-15 13:02:16,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10399.78 MB 2025-02-15 13:02:16,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50266.56 MB 2025-02-15 13:02:18,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:02:18,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:02:18,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:02:18,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:18,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34323.82 MB 2025-02-15 13:02:18,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34854.66 MB 2025-02-15 13:02:18,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:02:18,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53682.90 MB 2025-02-15 13:02:18,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41238.40 MB 2025-02-15 13:02:18,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12444.50 MB 2025-02-15 13:02:18,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38833.21 MB 2025-02-15 13:02:18,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:02:18,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:02:18,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:02:18,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:18,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34854.66 MB 2025-02-15 13:02:18,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36744.16 MB 2025-02-15 13:02:18,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:02:18,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41238.40 MB 2025-02-15 13:02:18,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41238.40 MB 2025-02-15 13:02:18,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:02:18,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38161.59 MB 2025-02-15 13:02:18,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:02:18,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:02:18,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:02:18,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:18,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36744.16 MB 2025-02-15 13:02:18,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38986.01 MB 2025-02-15 13:02:18,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:02:18,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41238.40 MB 2025-02-15 13:02:18,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46900.71 MB 2025-02-15 13:02:18,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:02:18,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44530.29 MB 2025-02-15 13:02:18,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:02:18,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:02:18,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:02:18,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:18,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34854.66 MB 2025-02-15 13:02:18,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38986.01 MB 2025-02-15 13:02:18,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:02:18,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41238.40 MB 2025-02-15 13:02:18,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46900.71 MB 2025-02-15 13:02:18,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:02:18,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44530.29 MB 2025-02-15 13:02:19,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:02:19,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:02:19,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:02:19,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:19,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39693.80 MB 2025-02-15 13:02:19,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40460.80 MB 2025-02-15 13:02:19,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:02:19,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46900.71 MB 2025-02-15 13:02:19,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47315.94 MB 2025-02-15 13:02:19,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:02:19,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41168.59 MB 2025-02-15 13:02:19,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:02:19,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:02:19,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:02:19,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:19,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40873.69 MB 2025-02-15 13:02:19,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41080.74 MB 2025-02-15 13:02:19,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.05 MB 2025-02-15 13:02:19,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47315.94 MB 2025-02-15 13:02:19,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47315.94 MB 2025-02-15 13:02:19,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:02:19,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41300.26 MB 2025-02-15 13:02:19,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:02:19,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:02:19,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.71 seconds 2025-02-15 13:02:19,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:19,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29493.82 MB 2025-02-15 13:02:19,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41281.62 MB 2025-02-15 13:02:19,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11787.80 MB 2025-02-15 13:02:19,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62243.47 MB 2025-02-15 13:02:19,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47315.94 MB 2025-02-15 13:02:19,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14927.53 MB 2025-02-15 13:02:19,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41300.26 MB 2025-02-15 13:02:19,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:02:19,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:02:19,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:02:19,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:19,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41281.62 MB 2025-02-15 13:02:19,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41381.99 MB 2025-02-15 13:02:19,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 13:02:19,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47315.94 MB 2025-02-15 13:02:19,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47315.94 MB 2025-02-15 13:02:19,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:02:19,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41984.20 MB 2025-02-15 13:02:19,451 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 13:02:19,451 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:02:19,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:02:19,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:02:19,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:02:19,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:02:19,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30756.55 MB 2025-02-15 13:02:19,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34946.93 MB 2025-02-15 13:02:19,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 13:02:19,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47315.94 MB 2025-02-15 13:02:19,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55698.26 MB 2025-02-15 13:02:19,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 13:02:19,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39137.04 MB 2025-02-15 13:02:19,616 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 13:02:19,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:02:19,623 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:02:19,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,624 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:02:19,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:02:19,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,625 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,625 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:02:19,631 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:02:19,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,632 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,632 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:02:19,632 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:02:19,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,633 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:02:19,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,633 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:02:19,633 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:02:19,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,634 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:02:19,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,637 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,638 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,639 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:02:19,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:02:19,645 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:09,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:09,154 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:09,161 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:03:09,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:09,163 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 953, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:03:09,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:09,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 953, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:03:24,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:03:24,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:03:24,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.92 seconds 2025-02-15 13:03:24,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:24,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32151.89 MB 2025-02-15 13:03:24,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35524.51 MB 2025-02-15 13:03:24,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3372.61 MB 2025-02-15 13:03:24,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62021.17 MB 2025-02-15 13:03:24,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42609.93 MB 2025-02-15 13:03:24,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19411.24 MB 2025-02-15 13:03:24,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44341.17 MB 2025-02-15 13:03:24,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:03:24,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:03:24,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:03:24,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:24,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35524.51 MB 2025-02-15 13:03:24,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33274.73 MB 2025-02-15 13:03:24,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2249.78 MB 2025-02-15 13:03:24,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42609.93 MB 2025-02-15 13:03:24,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50650.42 MB 2025-02-15 13:03:24,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8040.48 MB 2025-02-15 13:03:24,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46372.75 MB 2025-02-15 13:03:26,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:03:26,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:03:26,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:03:26,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33274.73 MB 2025-02-15 13:03:26,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33805.57 MB 2025-02-15 13:03:26,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:03:26,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50650.42 MB 2025-02-15 13:03:26,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40651.19 MB 2025-02-15 13:03:26,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9999.22 MB 2025-02-15 13:03:26,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37784.12 MB 2025-02-15 13:03:26,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:03:26,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:03:26,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:03:26,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33805.57 MB 2025-02-15 13:03:26,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35695.07 MB 2025-02-15 13:03:26,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:03:26,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40651.19 MB 2025-02-15 13:03:26,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40651.19 MB 2025-02-15 13:03:26,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:03:26,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37112.49 MB 2025-02-15 13:03:26,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:03:26,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:03:26,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:03:26,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35695.07 MB 2025-02-15 13:03:26,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37936.92 MB 2025-02-15 13:03:26,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:03:26,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40651.19 MB 2025-02-15 13:03:26,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46313.50 MB 2025-02-15 13:03:26,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:03:26,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43481.20 MB 2025-02-15 13:03:26,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:03:26,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:03:26,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:03:26,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33805.57 MB 2025-02-15 13:03:26,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37936.92 MB 2025-02-15 13:03:26,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:03:26,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40651.19 MB 2025-02-15 13:03:26,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46313.50 MB 2025-02-15 13:03:26,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:03:26,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43481.20 MB 2025-02-15 13:03:26,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:03:26,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:03:26,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:03:26,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38644.71 MB 2025-02-15 13:03:26,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39411.71 MB 2025-02-15 13:03:26,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:03:26,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46313.50 MB 2025-02-15 13:03:26,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46728.74 MB 2025-02-15 13:03:26,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:03:26,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40119.50 MB 2025-02-15 13:03:26,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:03:26,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:03:26,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:03:26,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39824.60 MB 2025-02-15 13:03:26,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40030.29 MB 2025-02-15 13:03:26,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.69 MB 2025-02-15 13:03:26,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46728.74 MB 2025-02-15 13:03:26,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46728.74 MB 2025-02-15 13:03:26,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:03:26,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40247.43 MB 2025-02-15 13:03:26,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:03:26,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:03:26,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.33 seconds 2025-02-15 13:03:26,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28831.56 MB 2025-02-15 13:03:26,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40230.45 MB 2025-02-15 13:03:26,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11398.89 MB 2025-02-15 13:03:26,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62021.17 MB 2025-02-15 13:03:26,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46728.74 MB 2025-02-15 13:03:26,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15292.43 MB 2025-02-15 13:03:26,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40247.43 MB 2025-02-15 13:03:26,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:03:26,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:03:26,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:03:26,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40230.45 MB 2025-02-15 13:03:26,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40330.46 MB 2025-02-15 13:03:26,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.01 MB 2025-02-15 13:03:26,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46728.74 MB 2025-02-15 13:03:26,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46728.74 MB 2025-02-15 13:03:26,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:03:26,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40930.54 MB 2025-02-15 13:03:26,786 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 13:03:26,786 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:03:26,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:03:26,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:03:26,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:03:26,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:03:26,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30093.58 MB 2025-02-15 13:03:26,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34269.08 MB 2025-02-15 13:03:26,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.50 MB 2025-02-15 13:03:26,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46728.74 MB 2025-02-15 13:03:26,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57170.46 MB 2025-02-15 13:03:26,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10441.72 MB 2025-02-15 13:03:26,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38444.51 MB 2025-02-15 13:03:26,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 13:03:26,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,953 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:03:26,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:03:26,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,960 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:03:26,960 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:03:26,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,961 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,962 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:26,967 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:03:26,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,968 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,968 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:26,969 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:03:26,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,969 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:26,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,969 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:03:26,970 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:03:26,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,970 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:03:26,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,974 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,975 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,976 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:03:26,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:03:26,982 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:22,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:22,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:22,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:04:22,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:22,903 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1065, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:04:22,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:22,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1065, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:04:39,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:04:39,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:04:39,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.42 seconds 2025-02-15 13:04:39,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:39,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33054.10 MB 2025-02-15 13:04:39,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36823.08 MB 2025-02-15 13:04:39,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3768.98 MB 2025-02-15 13:04:39,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63615.01 MB 2025-02-15 13:04:39,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41043.36 MB 2025-02-15 13:04:39,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22571.65 MB 2025-02-15 13:04:39,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45697.17 MB 2025-02-15 13:04:39,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:04:39,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:04:39,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:04:39,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:39,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36823.08 MB 2025-02-15 13:04:39,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33979.81 MB 2025-02-15 13:04:39,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2843.27 MB 2025-02-15 13:04:39,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41043.36 MB 2025-02-15 13:04:39,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50585.40 MB 2025-02-15 13:04:39,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9542.04 MB 2025-02-15 13:04:39,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48465.22 MB 2025-02-15 13:04:41,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:04:41,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:04:41,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:04:41,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33979.81 MB 2025-02-15 13:04:41,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34510.65 MB 2025-02-15 13:04:41,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:04:41,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50585.40 MB 2025-02-15 13:04:41,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39397.10 MB 2025-02-15 13:04:41,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11188.31 MB 2025-02-15 13:04:41,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38489.19 MB 2025-02-15 13:04:41,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:04:41,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:04:41,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:04:41,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34510.65 MB 2025-02-15 13:04:41,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36400.14 MB 2025-02-15 13:04:41,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:04:41,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39397.10 MB 2025-02-15 13:04:41,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41284.53 MB 2025-02-15 13:04:41,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:04:41,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37817.57 MB 2025-02-15 13:04:41,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:04:41,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:04:41,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:04:41,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36400.14 MB 2025-02-15 13:04:41,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38642.00 MB 2025-02-15 13:04:41,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:04:41,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41284.53 MB 2025-02-15 13:04:41,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47418.70 MB 2025-02-15 13:04:41,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:04:41,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44186.28 MB 2025-02-15 13:04:41,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:04:41,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:04:41,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:04:41,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34510.65 MB 2025-02-15 13:04:41,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38642.00 MB 2025-02-15 13:04:41,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:04:41,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39397.10 MB 2025-02-15 13:04:41,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47418.70 MB 2025-02-15 13:04:41,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 13:04:41,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44186.28 MB 2025-02-15 13:04:41,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:04:41,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:04:41,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:04:41,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39349.79 MB 2025-02-15 13:04:41,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40116.79 MB 2025-02-15 13:04:41,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:04:41,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47418.70 MB 2025-02-15 13:04:41,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47833.94 MB 2025-02-15 13:04:41,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:04:41,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40824.58 MB 2025-02-15 13:04:41,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:04:41,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:04:41,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:04:41,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40529.68 MB 2025-02-15 13:04:41,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40736.76 MB 2025-02-15 13:04:41,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.09 MB 2025-02-15 13:04:41,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47833.94 MB 2025-02-15 13:04:41,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47833.94 MB 2025-02-15 13:04:41,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:04:41,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40954.50 MB 2025-02-15 13:04:41,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:04:41,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:04:41,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.84 seconds 2025-02-15 13:04:41,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:41,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29343.56 MB 2025-02-15 13:04:41,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40937.81 MB 2025-02-15 13:04:41,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11594.26 MB 2025-02-15 13:04:41,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63615.01 MB 2025-02-15 13:04:41,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47833.94 MB 2025-02-15 13:04:41,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15781.07 MB 2025-02-15 13:04:41,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40954.50 MB 2025-02-15 13:04:42,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:04:42,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:04:42,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:04:42,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:42,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40937.81 MB 2025-02-15 13:04:42,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41038.27 MB 2025-02-15 13:04:42,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.45 MB 2025-02-15 13:04:42,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47833.94 MB 2025-02-15 13:04:42,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47833.94 MB 2025-02-15 13:04:42,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:04:42,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41640.99 MB 2025-02-15 13:04:42,034 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 13:04:42,035 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:04:42,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:04:42,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:04:42,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:04:42,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:04:42,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30606.45 MB 2025-02-15 13:04:42,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34800.76 MB 2025-02-15 13:04:42,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-15 13:04:42,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47833.94 MB 2025-02-15 13:04:42,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56222.55 MB 2025-02-15 13:04:42,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 13:04:42,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38995.06 MB 2025-02-15 13:04:42,203 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 13:04:42,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,204 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:04:42,210 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:04:42,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,211 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:04:42,211 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:04:42,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,212 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,212 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:42,218 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:04:42,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,219 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,219 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:42,219 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:04:42,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,220 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:42,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,220 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:04:42,220 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:04:42,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,221 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:04:42,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,224 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,225 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,226 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:04:42,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:04:42,232 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:28,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:28,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:28,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:05:28,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:28,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:05:28,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:28,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:05:47,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:05:47,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:05:47,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.20 seconds 2025-02-15 13:05:47,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:47,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34381.37 MB 2025-02-15 13:05:47,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38762.58 MB 2025-02-15 13:05:47,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-15 13:05:47,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62788.73 MB 2025-02-15 13:05:47,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48077.21 MB 2025-02-15 13:05:47,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14711.52 MB 2025-02-15 13:05:47,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47703.11 MB 2025-02-15 13:05:47,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:05:47,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:05:47,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:05:47,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:47,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38762.58 MB 2025-02-15 13:05:47,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34999.90 MB 2025-02-15 13:05:47,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-15 13:05:47,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48077.21 MB 2025-02-15 13:05:47,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56740.54 MB 2025-02-15 13:05:47,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8663.33 MB 2025-02-15 13:05:47,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51786.18 MB 2025-02-15 13:05:49,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:05:49,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:05:49,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:05:49,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34999.90 MB 2025-02-15 13:05:49,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35530.75 MB 2025-02-15 13:05:49,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:05:49,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56740.54 MB 2025-02-15 13:05:49,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43694.16 MB 2025-02-15 13:05:49,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13046.38 MB 2025-02-15 13:05:49,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39509.29 MB 2025-02-15 13:05:49,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:05:49,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:05:49,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:05:49,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35530.75 MB 2025-02-15 13:05:49,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37420.07 MB 2025-02-15 13:05:49,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.33 MB 2025-02-15 13:05:49,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43694.16 MB 2025-02-15 13:05:49,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43694.16 MB 2025-02-15 13:05:49,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:05:49,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38837.50 MB 2025-02-15 13:05:49,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:05:49,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:05:49,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:05:49,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37420.07 MB 2025-02-15 13:05:49,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39661.93 MB 2025-02-15 13:05:49,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:05:49,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43694.16 MB 2025-02-15 13:05:49,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48412.75 MB 2025-02-15 13:05:49,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:05:49,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45206.21 MB 2025-02-15 13:05:49,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:05:49,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:05:49,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:05:49,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35530.75 MB 2025-02-15 13:05:49,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39661.93 MB 2025-02-15 13:05:49,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.18 MB 2025-02-15 13:05:49,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43694.16 MB 2025-02-15 13:05:49,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48412.75 MB 2025-02-15 13:05:49,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:05:49,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45206.21 MB 2025-02-15 13:05:49,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:05:49,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:05:49,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:05:49,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40369.72 MB 2025-02-15 13:05:49,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41136.72 MB 2025-02-15 13:05:49,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:05:49,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48412.75 MB 2025-02-15 13:05:49,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48827.99 MB 2025-02-15 13:05:49,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:05:49,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41844.51 MB 2025-02-15 13:05:49,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:05:49,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:05:49,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:05:49,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41549.61 MB 2025-02-15 13:05:49,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41756.21 MB 2025-02-15 13:05:49,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.60 MB 2025-02-15 13:05:49,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48827.99 MB 2025-02-15 13:05:49,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48827.99 MB 2025-02-15 13:05:49,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:05:49,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41978.88 MB 2025-02-15 13:05:49,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:05:49,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:05:49,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.62 seconds 2025-02-15 13:05:49,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:49,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30068.08 MB 2025-02-15 13:05:49,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41957.19 MB 2025-02-15 13:05:49,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11889.11 MB 2025-02-15 13:05:49,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62788.73 MB 2025-02-15 13:05:49,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48827.99 MB 2025-02-15 13:05:49,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13960.74 MB 2025-02-15 13:05:49,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41978.88 MB 2025-02-15 13:05:50,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:05:50,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:05:50,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:05:50,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:50,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41957.19 MB 2025-02-15 13:05:50,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42057.60 MB 2025-02-15 13:05:50,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.42 MB 2025-02-15 13:05:50,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48827.99 MB 2025-02-15 13:05:50,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48827.99 MB 2025-02-15 13:05:50,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:05:50,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42660.11 MB 2025-02-15 13:05:50,028 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-15 13:05:50,028 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:05:50,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:05:50,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:05:50,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:05:50,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:05:50,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31330.90 MB 2025-02-15 13:05:50,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35523.33 MB 2025-02-15 13:05:50,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.43 MB 2025-02-15 13:05:50,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48827.99 MB 2025-02-15 13:05:50,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57214.50 MB 2025-02-15 13:05:50,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-15 13:05:50,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39715.54 MB 2025-02-15 13:05:50,195 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-15 13:05:50,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,197 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,197 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:05:50,202 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:05:50,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,203 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:05:50,203 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:05:50,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,204 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,205 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:50,210 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:05:50,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,211 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,211 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:50,211 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:05:50,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,212 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:50,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,212 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:05:50,212 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:05:50,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,213 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:05:50,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,217 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,217 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,218 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:05:50,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:05:50,224 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:31,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:31,398 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:31,403 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:06:31,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:31,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1140, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:06:31,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:31,405 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1140, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:06:49,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:06:49,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:06:49,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.68 seconds 2025-02-15 13:06:49,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:49,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33820.29 MB 2025-02-15 13:06:49,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37855.21 MB 2025-02-15 13:06:49,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4034.92 MB 2025-02-15 13:06:49,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63902.32 MB 2025-02-15 13:06:49,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43656.41 MB 2025-02-15 13:06:49,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20245.91 MB 2025-02-15 13:06:49,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46689.04 MB 2025-02-15 13:06:49,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:06:49,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:06:49,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:06:49,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:49,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37855.21 MB 2025-02-15 13:06:49,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34613.28 MB 2025-02-15 13:06:49,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3241.93 MB 2025-02-15 13:06:49,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43656.41 MB 2025-02-15 13:06:49,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53768.88 MB 2025-02-15 13:06:49,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10112.47 MB 2025-02-15 13:06:49,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50067.72 MB 2025-02-15 13:06:51,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:06:51,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:06:51,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:06:51,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34613.28 MB 2025-02-15 13:06:51,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35144.12 MB 2025-02-15 13:06:51,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:06:51,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53768.88 MB 2025-02-15 13:06:51,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41745.91 MB 2025-02-15 13:06:51,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12022.97 MB 2025-02-15 13:06:51,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39122.67 MB 2025-02-15 13:06:51,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:06:51,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:06:51,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:06:51,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35144.12 MB 2025-02-15 13:06:51,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37033.62 MB 2025-02-15 13:06:51,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:06:51,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41745.91 MB 2025-02-15 13:06:51,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42689.63 MB 2025-02-15 13:06:51,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 13:06:51,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38451.04 MB 2025-02-15 13:06:51,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:06:51,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:06:51,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:06:51,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37033.62 MB 2025-02-15 13:06:51,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39275.47 MB 2025-02-15 13:06:51,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:06:51,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42689.63 MB 2025-02-15 13:06:51,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47880.08 MB 2025-02-15 13:06:51,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:06:51,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44819.75 MB 2025-02-15 13:06:51,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:06:51,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:06:51,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:06:51,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35144.12 MB 2025-02-15 13:06:51,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39275.47 MB 2025-02-15 13:06:51,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:06:51,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41745.91 MB 2025-02-15 13:06:51,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47880.08 MB 2025-02-15 13:06:51,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:06:51,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44819.75 MB 2025-02-15 13:06:51,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:06:51,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:06:51,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:06:51,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39983.26 MB 2025-02-15 13:06:51,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40750.26 MB 2025-02-15 13:06:51,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:06:51,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47880.08 MB 2025-02-15 13:06:51,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48295.31 MB 2025-02-15 13:06:51,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:06:51,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41458.05 MB 2025-02-15 13:06:51,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:06:51,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:06:51,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:06:51,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41163.15 MB 2025-02-15 13:06:51,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41368.94 MB 2025-02-15 13:06:51,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.79 MB 2025-02-15 13:06:51,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48295.31 MB 2025-02-15 13:06:51,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48295.31 MB 2025-02-15 13:06:51,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:06:51,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41584.04 MB 2025-02-15 13:06:51,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:06:51,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:06:51,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.11 seconds 2025-02-15 13:06:51,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29848.44 MB 2025-02-15 13:06:51,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41569.20 MB 2025-02-15 13:06:51,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11720.76 MB 2025-02-15 13:06:51,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63902.32 MB 2025-02-15 13:06:51,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48295.31 MB 2025-02-15 13:06:51,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15607.01 MB 2025-02-15 13:06:51,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41584.04 MB 2025-02-15 13:06:51,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:06:51,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:06:51,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:06:51,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41569.20 MB 2025-02-15 13:06:51,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41669.26 MB 2025-02-15 13:06:51,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.06 MB 2025-02-15 13:06:51,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48295.31 MB 2025-02-15 13:06:51,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48295.31 MB 2025-02-15 13:06:51,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:06:51,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42269.63 MB 2025-02-15 13:06:51,805 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 13:06:51,805 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:06:51,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:06:51,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:06:51,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:06:51,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:06:51,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31110.54 MB 2025-02-15 13:06:51,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35288.10 MB 2025-02-15 13:06:51,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.56 MB 2025-02-15 13:06:51,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48295.31 MB 2025-02-15 13:06:51,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56652.46 MB 2025-02-15 13:06:51,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-15 13:06:51,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39465.63 MB 2025-02-15 13:06:51,973 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 13:06:51,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,974 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:51,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,975 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:06:51,980 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:06:51,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,981 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:06:51,981 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:06:51,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,982 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:51,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,982 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:51,988 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:06:51,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,989 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:51,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,989 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:51,989 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:06:51,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:51,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:06:51,990 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:06:51,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,991 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:06:51,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,996 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:51,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,997 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:51,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:51,998 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:06:52,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:06:52,004 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:07:41,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:07:41,280 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:07:41,286 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:07:41,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:07:41,287 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1042, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:07:41,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:07:41,288 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1042, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:07:57,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:07:57,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:07:57,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.20 seconds 2025-02-15 13:07:57,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:57,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33259.16 MB 2025-02-15 13:07:57,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36946.74 MB 2025-02-15 13:07:57,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3687.58 MB 2025-02-15 13:07:57,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63461.92 MB 2025-02-15 13:07:57,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43432.02 MB 2025-02-15 13:07:57,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20029.90 MB 2025-02-15 13:07:57,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45901.42 MB 2025-02-15 13:07:57,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:07:57,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:07:57,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:07:57,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:57,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36946.74 MB 2025-02-15 13:07:57,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34225.56 MB 2025-02-15 13:07:57,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2721.18 MB 2025-02-15 13:07:57,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43432.02 MB 2025-02-15 13:07:57,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52871.30 MB 2025-02-15 13:07:57,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-15 13:07:57,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48484.19 MB 2025-02-15 13:07:59,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:07:59,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:07:59,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:07:59,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34225.56 MB 2025-02-15 13:07:59,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34756.40 MB 2025-02-15 13:07:59,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:07:59,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52871.30 MB 2025-02-15 13:07:59,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41867.54 MB 2025-02-15 13:07:59,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11003.76 MB 2025-02-15 13:07:59,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38734.95 MB 2025-02-15 13:07:59,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:07:59,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:07:59,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:07:59,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34756.40 MB 2025-02-15 13:07:59,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36645.89 MB 2025-02-15 13:07:59,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:07:59,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41867.54 MB 2025-02-15 13:07:59,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41867.54 MB 2025-02-15 13:07:59,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:07:59,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38063.32 MB 2025-02-15 13:07:59,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:07:59,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:07:59,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:07:59,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36645.89 MB 2025-02-15 13:07:59,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38887.75 MB 2025-02-15 13:07:59,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:07:59,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41867.54 MB 2025-02-15 13:07:59,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47529.85 MB 2025-02-15 13:07:59,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:07:59,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44432.03 MB 2025-02-15 13:07:59,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:07:59,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:07:59,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:07:59,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34756.40 MB 2025-02-15 13:07:59,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38887.75 MB 2025-02-15 13:07:59,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:07:59,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41867.54 MB 2025-02-15 13:07:59,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47529.85 MB 2025-02-15 13:07:59,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:07:59,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44432.03 MB 2025-02-15 13:07:59,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:07:59,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:07:59,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:07:59,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39595.54 MB 2025-02-15 13:07:59,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40362.54 MB 2025-02-15 13:07:59,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:07:59,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47529.85 MB 2025-02-15 13:07:59,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47945.09 MB 2025-02-15 13:07:59,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:07:59,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41070.33 MB 2025-02-15 13:07:59,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:07:59,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:07:59,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:07:59,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40775.43 MB 2025-02-15 13:07:59,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40981.30 MB 2025-02-15 13:07:59,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.87 MB 2025-02-15 13:07:59,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47945.09 MB 2025-02-15 13:07:59,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47945.09 MB 2025-02-15 13:07:59,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:07:59,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41206.11 MB 2025-02-15 13:07:59,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:07:59,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:07:59,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.62 seconds 2025-02-15 13:07:59,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:07:59,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29628.74 MB 2025-02-15 13:07:59,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41182.08 MB 2025-02-15 13:07:59,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11553.33 MB 2025-02-15 13:07:59,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63461.92 MB 2025-02-15 13:07:59,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47945.09 MB 2025-02-15 13:07:59,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15516.83 MB 2025-02-15 13:07:59,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41206.11 MB 2025-02-15 13:08:00,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:08:00,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:08:00,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:08:00,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:08:00,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41182.08 MB 2025-02-15 13:08:00,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41282.40 MB 2025-02-15 13:08:00,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-15 13:08:00,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47945.09 MB 2025-02-15 13:08:00,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47945.09 MB 2025-02-15 13:08:00,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:08:00,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41884.31 MB 2025-02-15 13:08:00,194 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 13:08:00,194 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:08:00,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:08:00,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:08:00,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:08:00,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:08:00,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30891.37 MB 2025-02-15 13:08:00,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35079.70 MB 2025-02-15 13:08:00,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-15 13:08:00,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47945.09 MB 2025-02-15 13:08:00,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58418.27 MB 2025-02-15 13:08:00,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 13:08:00,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39267.71 MB 2025-02-15 13:08:00,360 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 13:08:00,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,362 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,363 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:08:00,367 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:08:00,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,368 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:08:00,368 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:08:00,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,369 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,370 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:00,376 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:08:00,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,376 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,377 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:00,377 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:08:00,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,377 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:00,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,378 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:08:00,378 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:08:00,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,378 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:00,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,381 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,382 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,383 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:08:00,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:00,389 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:44,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:44,095 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:08:44,100 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:08:44,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:44,101 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:08:44,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:08:44,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:09:02,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:09:02,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:09:02,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.21 seconds 2025-02-15 13:09:02,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:02,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34300.73 MB 2025-02-15 13:09:02,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38455.45 MB 2025-02-15 13:09:02,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4154.72 MB 2025-02-15 13:09:02,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65349.35 MB 2025-02-15 13:09:02,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46110.08 MB 2025-02-15 13:09:02,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19239.27 MB 2025-02-15 13:09:02,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47395.98 MB 2025-02-15 13:09:02,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:09:02,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:09:02,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:09:02,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:02,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38455.45 MB 2025-02-15 13:09:02,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35032.51 MB 2025-02-15 13:09:02,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3422.94 MB 2025-02-15 13:09:02,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46110.08 MB 2025-02-15 13:09:02,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54322.53 MB 2025-02-15 13:09:02,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8212.45 MB 2025-02-15 13:09:02,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50891.30 MB 2025-02-15 13:09:04,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:09:04,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:09:04,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:09:04,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35032.51 MB 2025-02-15 13:09:04,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35563.35 MB 2025-02-15 13:09:04,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:09:04,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54322.53 MB 2025-02-15 13:09:04,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 13:09:04,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12369.00 MB 2025-02-15 13:09:04,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39541.90 MB 2025-02-15 13:09:04,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:09:04,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:09:04,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:09:04,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35563.35 MB 2025-02-15 13:09:04,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37452.85 MB 2025-02-15 13:09:04,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:09:04,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 13:09:04,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41953.53 MB 2025-02-15 13:09:04,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:09:04,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38870.27 MB 2025-02-15 13:09:04,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:09:04,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:09:04,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:09:04,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37452.85 MB 2025-02-15 13:09:04,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39694.70 MB 2025-02-15 13:09:04,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:09:04,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 13:09:04,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47615.84 MB 2025-02-15 13:09:04,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:09:04,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45238.98 MB 2025-02-15 13:09:04,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:09:04,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:09:04,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:09:04,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35563.35 MB 2025-02-15 13:09:04,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39694.70 MB 2025-02-15 13:09:04,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:09:04,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41953.53 MB 2025-02-15 13:09:04,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47615.84 MB 2025-02-15 13:09:04,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:09:04,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45238.98 MB 2025-02-15 13:09:04,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:09:04,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:09:04,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:09:04,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40402.49 MB 2025-02-15 13:09:04,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41169.49 MB 2025-02-15 13:09:04,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:09:04,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47615.84 MB 2025-02-15 13:09:04,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48031.07 MB 2025-02-15 13:09:04,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:09:04,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41877.28 MB 2025-02-15 13:09:04,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:09:04,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:09:04,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:09:04,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41582.38 MB 2025-02-15 13:09:04,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41787.58 MB 2025-02-15 13:09:04,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.20 MB 2025-02-15 13:09:04,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48031.07 MB 2025-02-15 13:09:04,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48031.07 MB 2025-02-15 13:09:04,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:09:04,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42006.08 MB 2025-02-15 13:09:04,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:09:04,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:09:04,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.64 seconds 2025-02-15 13:09:04,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:04,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30210.42 MB 2025-02-15 13:09:04,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41988.45 MB 2025-02-15 13:09:04,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11778.04 MB 2025-02-15 13:09:04,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65349.35 MB 2025-02-15 13:09:04,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48031.07 MB 2025-02-15 13:09:04,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17318.28 MB 2025-02-15 13:09:04,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42006.08 MB 2025-02-15 13:09:05,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:09:05,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:09:05,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:09:05,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:05,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41988.45 MB 2025-02-15 13:09:05,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42088.82 MB 2025-02-15 13:09:05,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 13:09:05,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48031.07 MB 2025-02-15 13:09:05,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48031.07 MB 2025-02-15 13:09:05,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:09:05,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42691.03 MB 2025-02-15 13:09:05,027 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 13:09:05,027 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:09:05,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:09:05,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:09:05,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:09:05,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:09:05,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31473.14 MB 2025-02-15 13:09:05,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35663.52 MB 2025-02-15 13:09:05,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 13:09:05,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48031.07 MB 2025-02-15 13:09:05,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56413.39 MB 2025-02-15 13:09:05,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-15 13:09:05,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39853.63 MB 2025-02-15 13:09:05,194 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 13:09:05,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,196 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,196 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:09:05,201 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:09:05,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,202 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:09:05,202 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:09:05,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,203 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,204 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:09:05,209 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:09:05,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,210 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,211 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:09:05,211 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:09:05,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,211 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:09:05,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,212 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:09:05,212 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:09:05,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,212 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:09:05,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,216 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,216 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,217 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:09:05,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:09:05,224 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:01,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:01,325 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:01,333 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:10:01,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:01,335 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:10:01,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:01,337 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:10:19,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:10:19,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:10:19,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.13 seconds 2025-02-15 13:10:19,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:19,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34331.91 MB 2025-02-15 13:10:19,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38440.63 MB 2025-02-15 13:10:19,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4108.71 MB 2025-02-15 13:10:19,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63466.11 MB 2025-02-15 13:10:19,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46185.58 MB 2025-02-15 13:10:19,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17280.53 MB 2025-02-15 13:10:19,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47427.16 MB 2025-02-15 13:10:19,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:10:19,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:10:19,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:10:19,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:19,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38440.63 MB 2025-02-15 13:10:19,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35086.70 MB 2025-02-15 13:10:19,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3353.93 MB 2025-02-15 13:10:19,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46185.58 MB 2025-02-15 13:10:19,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54343.50 MB 2025-02-15 13:10:19,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8157.92 MB 2025-02-15 13:10:19,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50820.00 MB 2025-02-15 13:10:21,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:10:21,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:10:21,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:10:21,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35086.70 MB 2025-02-15 13:10:21,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35617.54 MB 2025-02-15 13:10:21,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:10:21,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54343.50 MB 2025-02-15 13:10:21,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39980.11 MB 2025-02-15 13:10:21,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14363.39 MB 2025-02-15 13:10:21,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39597.13 MB 2025-02-15 13:10:21,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:10:21,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:10:21,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:10:21,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35617.54 MB 2025-02-15 13:10:21,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37507.04 MB 2025-02-15 13:10:21,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:10:21,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39980.11 MB 2025-02-15 13:10:21,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41867.54 MB 2025-02-15 13:10:21,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:10:21,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38924.47 MB 2025-02-15 13:10:21,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:10:21,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:10:21,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:10:21,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37507.04 MB 2025-02-15 13:10:21,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39748.89 MB 2025-02-15 13:10:21,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:10:21,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41867.54 MB 2025-02-15 13:10:21,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48001.71 MB 2025-02-15 13:10:21,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:10:21,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45293.17 MB 2025-02-15 13:10:21,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:10:21,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:10:21,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:10:21,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35617.54 MB 2025-02-15 13:10:21,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39748.89 MB 2025-02-15 13:10:21,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:10:21,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39980.11 MB 2025-02-15 13:10:21,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48001.71 MB 2025-02-15 13:10:21,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 13:10:21,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45293.17 MB 2025-02-15 13:10:21,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:10:21,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:10:21,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:10:21,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40456.68 MB 2025-02-15 13:10:21,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41223.68 MB 2025-02-15 13:10:21,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:10:21,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48001.71 MB 2025-02-15 13:10:21,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48416.95 MB 2025-02-15 13:10:21,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:10:21,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41931.47 MB 2025-02-15 13:10:21,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:10:21,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:10:21,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:10:21,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41636.57 MB 2025-02-15 13:10:21,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41841.56 MB 2025-02-15 13:10:21,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.99 MB 2025-02-15 13:10:21,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48416.95 MB 2025-02-15 13:10:21,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48416.95 MB 2025-02-15 13:10:21,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:10:21,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42066.23 MB 2025-02-15 13:10:21,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:10:21,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:10:21,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.55 seconds 2025-02-15 13:10:21,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:21,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30286.90 MB 2025-02-15 13:10:21,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42041.89 MB 2025-02-15 13:10:21,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11755.00 MB 2025-02-15 13:10:21,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63466.11 MB 2025-02-15 13:10:21,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48416.95 MB 2025-02-15 13:10:21,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15049.16 MB 2025-02-15 13:10:21,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42066.23 MB 2025-02-15 13:10:22,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:10:22,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:10:22,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:10:22,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:22,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42041.89 MB 2025-02-15 13:10:22,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42141.99 MB 2025-02-15 13:10:22,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.10 MB 2025-02-15 13:10:22,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48416.95 MB 2025-02-15 13:10:22,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48416.95 MB 2025-02-15 13:10:22,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:10:22,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42742.58 MB 2025-02-15 13:10:22,202 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 13:10:22,203 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:10:22,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:10:22,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:10:22,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 13:10:22,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:10:22,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31549.08 MB 2025-02-15 13:10:22,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35728.71 MB 2025-02-15 13:10:22,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-15 13:10:22,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48416.95 MB 2025-02-15 13:10:22,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56776.20 MB 2025-02-15 13:10:22,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 13:10:22,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39908.33 MB 2025-02-15 13:10:22,376 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 13:10:22,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,377 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,378 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:10:22,383 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:10:22,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,384 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:10:22,384 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:10:22,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,385 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,386 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:22,391 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:10:22,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,392 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,392 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:22,392 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:10:22,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,393 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:22,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,393 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:10:22,393 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:10:22,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,394 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:10:22,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,398 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,399 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,399 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:10:22,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:10:22,405 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:00,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:00,215 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:00,220 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:11:00,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:00,221 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:11:00,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:00,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:11:17,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:11:17,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:11:17,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.61 seconds 2025-02-15 13:11:17,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:17,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34279.48 MB 2025-02-15 13:11:17,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38299.72 MB 2025-02-15 13:11:17,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-15 13:11:17,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63950.55 MB 2025-02-15 13:11:17,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44121.98 MB 2025-02-15 13:11:17,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19828.57 MB 2025-02-15 13:11:17,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47148.24 MB 2025-02-15 13:11:17,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:11:17,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:11:17,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:11:17,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:17,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38299.72 MB 2025-02-15 13:11:17,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35079.56 MB 2025-02-15 13:11:17,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3220.17 MB 2025-02-15 13:11:17,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44121.98 MB 2025-02-15 13:11:17,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54226.06 MB 2025-02-15 13:11:17,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10104.08 MB 2025-02-15 13:11:17,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50505.22 MB 2025-02-15 13:11:19,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:11:19,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:11:19,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:11:19,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:19,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35079.56 MB 2025-02-15 13:11:19,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35610.40 MB 2025-02-15 13:11:19,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:11:19,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54226.06 MB 2025-02-15 13:11:19,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42226.16 MB 2025-02-15 13:11:19,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11999.90 MB 2025-02-15 13:11:19,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39588.95 MB 2025-02-15 13:11:19,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:11:19,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:11:19,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:11:19,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:19,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35610.40 MB 2025-02-15 13:11:19,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37499.89 MB 2025-02-15 13:11:19,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:11:19,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42226.16 MB 2025-02-15 13:11:19,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42226.16 MB 2025-02-15 13:11:19,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:11:19,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38917.32 MB 2025-02-15 13:11:20,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:11:20,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:11:20,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 13:11:20,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37499.89 MB 2025-02-15 13:11:20,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39741.75 MB 2025-02-15 13:11:20,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:11:20,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42226.16 MB 2025-02-15 13:11:20,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47888.47 MB 2025-02-15 13:11:20,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:11:20,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45286.03 MB 2025-02-15 13:11:20,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:11:20,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:11:20,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:11:20,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35610.40 MB 2025-02-15 13:11:20,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39741.75 MB 2025-02-15 13:11:20,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:11:20,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42226.16 MB 2025-02-15 13:11:20,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47888.47 MB 2025-02-15 13:11:20,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:11:20,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45286.03 MB 2025-02-15 13:11:20,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:11:20,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:11:20,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:11:20,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40449.54 MB 2025-02-15 13:11:20,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41216.54 MB 2025-02-15 13:11:20,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:11:20,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47888.47 MB 2025-02-15 13:11:20,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48303.70 MB 2025-02-15 13:11:20,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:11:20,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41924.33 MB 2025-02-15 13:11:20,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:11:20,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:11:20,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:11:20,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41629.43 MB 2025-02-15 13:11:20,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41835.41 MB 2025-02-15 13:11:20,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.98 MB 2025-02-15 13:11:20,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48303.70 MB 2025-02-15 13:11:20,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48303.70 MB 2025-02-15 13:11:20,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:11:20,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42060.00 MB 2025-02-15 13:11:20,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:11:20,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:11:20,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.19 seconds 2025-02-15 13:11:20,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30321.57 MB 2025-02-15 13:11:20,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42035.52 MB 2025-02-15 13:11:20,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11713.95 MB 2025-02-15 13:11:20,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63950.55 MB 2025-02-15 13:11:20,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48303.70 MB 2025-02-15 13:11:20,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15646.85 MB 2025-02-15 13:11:20,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42060.00 MB 2025-02-15 13:11:20,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:11:20,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:11:20,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 13:11:20,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42035.52 MB 2025-02-15 13:11:20,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42135.51 MB 2025-02-15 13:11:20,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.99 MB 2025-02-15 13:11:20,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48303.70 MB 2025-02-15 13:11:20,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48303.70 MB 2025-02-15 13:11:20,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:11:20,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42735.44 MB 2025-02-15 13:11:20,725 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 13:11:20,726 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:11:20,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:11:20,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:11:20,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 13:11:20,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:11:20,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31583.53 MB 2025-02-15 13:11:20,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35758.96 MB 2025-02-15 13:11:20,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.43 MB 2025-02-15 13:11:20,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48303.70 MB 2025-02-15 13:11:20,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56654.56 MB 2025-02-15 13:11:20,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 13:11:20,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39932.93 MB 2025-02-15 13:11:20,982 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 13:11:20,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:20,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:20,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:20,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:11:20,994 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:11:20,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:20,996 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:11:20,996 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:11:20,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:20,998 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:20,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:20,999 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:21,008 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:11:21,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,010 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:21,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,011 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:21,011 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:11:21,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,012 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:21,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,012 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:11:21,013 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:11:21,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,014 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:11:21,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,025 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:21,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,028 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:21,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,031 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:11:21,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:11:21,040 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:04,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:04,373 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:04,378 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:12:04,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:04,379 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 672, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:12:04,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:04,380 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 672, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:12:14,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:12:14,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:12:14,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.38 seconds 2025-02-15 13:12:14,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:14,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31168.03 MB 2025-02-15 13:12:14,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33546.20 MB 2025-02-15 13:12:14,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2378.17 MB 2025-02-15 13:12:14,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63950.55 MB 2025-02-15 13:12:14,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-15 13:12:14,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25782.39 MB 2025-02-15 13:12:14,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42451.34 MB 2025-02-15 13:12:14,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:12:14,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:12:14,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 13:12:14,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:14,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33546.20 MB 2025-02-15 13:12:14,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32789.14 MB 2025-02-15 13:12:14,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -757.06 MB 2025-02-15 13:12:14,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-15 13:12:14,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44937.77 MB 2025-02-15 13:12:14,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6769.61 MB 2025-02-15 13:12:14,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42414.84 MB 2025-02-15 13:12:16,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:12:16,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:12:16,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 13:12:16,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:16,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32789.14 MB 2025-02-15 13:12:16,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33319.98 MB 2025-02-15 13:12:16,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:12:16,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44937.77 MB 2025-02-15 13:12:16,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37914.41 MB 2025-02-15 13:12:16,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7023.36 MB 2025-02-15 13:12:16,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37299.57 MB 2025-02-15 13:12:16,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:12:16,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:12:16,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:12:16,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:16,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33319.98 MB 2025-02-15 13:12:16,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35209.47 MB 2025-02-15 13:12:16,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:12:16,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37914.41 MB 2025-02-15 13:12:16,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39801.85 MB 2025-02-15 13:12:16,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:12:16,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36626.90 MB 2025-02-15 13:12:16,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:12:16,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:12:16,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:12:16,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:16,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35209.47 MB 2025-02-15 13:12:16,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37451.33 MB 2025-02-15 13:12:16,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:12:16,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39801.85 MB 2025-02-15 13:12:16,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45936.02 MB 2025-02-15 13:12:16,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:12:16,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42995.61 MB 2025-02-15 13:12:16,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:12:16,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:12:16,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:12:16,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:16,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33319.98 MB 2025-02-15 13:12:16,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37451.33 MB 2025-02-15 13:12:16,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:12:16,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37914.41 MB 2025-02-15 13:12:16,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45936.02 MB 2025-02-15 13:12:16,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 13:12:16,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42995.61 MB 2025-02-15 13:12:17,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:12:17,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:12:17,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:12:17,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:17,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38159.12 MB 2025-02-15 13:12:17,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38926.12 MB 2025-02-15 13:12:17,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:12:17,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45936.02 MB 2025-02-15 13:12:17,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46351.25 MB 2025-02-15 13:12:17,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:12:17,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39633.91 MB 2025-02-15 13:12:17,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:12:17,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:12:17,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:12:17,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:17,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39339.01 MB 2025-02-15 13:12:17,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39543.27 MB 2025-02-15 13:12:17,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.26 MB 2025-02-15 13:12:17,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46351.25 MB 2025-02-15 13:12:17,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46351.25 MB 2025-02-15 13:12:17,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:12:17,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39742.27 MB 2025-02-15 13:12:17,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:12:17,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:12:17,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.74 seconds 2025-02-15 13:12:17,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:17,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28826.73 MB 2025-02-15 13:12:17,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39743.31 MB 2025-02-15 13:12:17,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10916.58 MB 2025-02-15 13:12:17,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63950.55 MB 2025-02-15 13:12:17,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46351.25 MB 2025-02-15 13:12:17,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17599.30 MB 2025-02-15 13:12:17,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39743.31 MB 2025-02-15 13:12:17,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:12:17,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:12:17,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:12:17,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:17,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39743.31 MB 2025-02-15 13:12:17,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39843.26 MB 2025-02-15 13:12:17,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.95 MB 2025-02-15 13:12:17,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46351.25 MB 2025-02-15 13:12:17,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46351.25 MB 2025-02-15 13:12:17,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:12:17,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40442.97 MB 2025-02-15 13:12:17,407 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 13:12:17,408 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 13:12:17,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:12:17,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:12:17,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:12:17,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:12:17,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30088.62 MB 2025-02-15 13:12:17,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34261.95 MB 2025-02-15 13:12:17,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.33 MB 2025-02-15 13:12:17,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46351.25 MB 2025-02-15 13:12:17,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54697.92 MB 2025-02-15 13:12:17,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 13:12:17,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38435.28 MB 2025-02-15 13:12:17,571 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 13:12:17,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,573 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:12:17,578 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:12:17,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,579 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:12:17,579 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 13:12:17,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,580 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,580 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:17,586 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:12:17,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,586 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,587 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:17,587 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:12:17,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,587 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:17,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,588 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:12:17,588 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:12:17,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,589 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:12:17,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,591 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,593 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,594 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:12:17,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:12:17,601 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:05,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:05,608 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:05,613 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:13:05,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:05,615 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 843, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:13:05,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:05,616 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 843, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:13:18,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:13:18,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:13:18,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.10 seconds 2025-02-15 13:13:18,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:18,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32482.99 MB 2025-02-15 13:13:18,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35467.23 MB 2025-02-15 13:13:18,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2984.25 MB 2025-02-15 13:13:18,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62115.55 MB 2025-02-15 13:13:18,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43069.21 MB 2025-02-15 13:13:18,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19046.33 MB 2025-02-15 13:13:18,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44445.77 MB 2025-02-15 13:13:18,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:13:18,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:13:18,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 13:13:18,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:18,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35467.23 MB 2025-02-15 13:13:18,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33800.47 MB 2025-02-15 13:13:18,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1666.77 MB 2025-02-15 13:13:18,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43069.21 MB 2025-02-15 13:13:18,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48500.83 MB 2025-02-15 13:13:18,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5431.62 MB 2025-02-15 13:13:18,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45058.37 MB 2025-02-15 13:13:20,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:13:20,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:13:20,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:13:20,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:20,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33800.47 MB 2025-02-15 13:13:20,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34331.31 MB 2025-02-15 13:13:20,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:13:20,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48500.83 MB 2025-02-15 13:13:20,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41500.54 MB 2025-02-15 13:13:20,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7000.29 MB 2025-02-15 13:13:20,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38309.85 MB 2025-02-15 13:13:20,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:13:20,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:13:20,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:13:20,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:20,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34331.31 MB 2025-02-15 13:13:20,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36220.80 MB 2025-02-15 13:13:20,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:13:20,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41500.54 MB 2025-02-15 13:13:20,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41500.54 MB 2025-02-15 13:13:20,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:13:20,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37638.23 MB 2025-02-15 13:13:20,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:13:20,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:13:20,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:13:20,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:20,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36220.80 MB 2025-02-15 13:13:20,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38462.66 MB 2025-02-15 13:13:20,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:13:20,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41500.54 MB 2025-02-15 13:13:20,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46690.99 MB 2025-02-15 13:13:20,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:13:20,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44006.94 MB 2025-02-15 13:13:20,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:13:20,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:13:20,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:13:20,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:20,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34331.31 MB 2025-02-15 13:13:20,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38462.66 MB 2025-02-15 13:13:20,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:13:20,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41500.54 MB 2025-02-15 13:13:20,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46690.99 MB 2025-02-15 13:13:20,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:13:20,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44006.94 MB 2025-02-15 13:13:21,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:13:21,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:13:21,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:13:21,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:21,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39170.45 MB 2025-02-15 13:13:21,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39937.45 MB 2025-02-15 13:13:21,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:13:21,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46690.99 MB 2025-02-15 13:13:21,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47106.23 MB 2025-02-15 13:13:21,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:13:21,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40645.24 MB 2025-02-15 13:13:21,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:13:21,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:13:21,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:13:21,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:21,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40350.34 MB 2025-02-15 13:13:21,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40556.32 MB 2025-02-15 13:13:21,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.98 MB 2025-02-15 13:13:21,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-15 13:13:21,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47106.23 MB 2025-02-15 13:13:21,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:13:21,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40749.15 MB 2025-02-15 13:13:21,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:13:21,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:13:21,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.48 seconds 2025-02-15 13:13:21,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:21,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29545.91 MB 2025-02-15 13:13:21,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40756.24 MB 2025-02-15 13:13:21,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11210.33 MB 2025-02-15 13:13:21,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62115.55 MB 2025-02-15 13:13:21,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47106.23 MB 2025-02-15 13:13:21,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15009.32 MB 2025-02-15 13:13:21,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40756.24 MB 2025-02-15 13:13:21,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:13:21,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:13:21,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:13:21,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:21,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40756.24 MB 2025-02-15 13:13:21,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40856.12 MB 2025-02-15 13:13:21,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.89 MB 2025-02-15 13:13:21,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-15 13:13:21,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47106.23 MB 2025-02-15 13:13:21,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:13:21,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41455.46 MB 2025-02-15 13:13:21,377 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 13:13:21,377 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:13:21,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:13:21,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:13:21,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:13:21,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:13:21,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30807.67 MB 2025-02-15 13:13:21,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34978.91 MB 2025-02-15 13:13:21,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4171.24 MB 2025-02-15 13:13:21,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-15 13:13:21,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55448.70 MB 2025-02-15 13:13:21,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 13:13:21,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39148.77 MB 2025-02-15 13:13:21,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 13:13:21,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:13:21,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:13:21,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:13:21,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:13:21,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,552 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,552 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:21,558 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:13:21,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,558 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,559 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:21,559 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:13:21,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,559 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:21,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,560 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:13:21,560 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:13:21,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,561 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:13:21,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,564 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,565 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,566 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:13:21,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:13:21,573 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:26,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:26,452 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:26,459 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:14:26,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:26,462 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:14:26,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:26,464 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:14:44,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:14:44,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:14:44,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.28 seconds 2025-02-15 13:14:44,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:44,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34916.56 MB 2025-02-15 13:14:44,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39075.22 MB 2025-02-15 13:14:44,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4158.65 MB 2025-02-15 13:14:44,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62987.96 MB 2025-02-15 13:14:44,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42945.48 MB 2025-02-15 13:14:44,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20042.48 MB 2025-02-15 13:14:44,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48012.62 MB 2025-02-15 13:14:44,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:14:44,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:14:44,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 13:14:44,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:44,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39075.22 MB 2025-02-15 13:14:44,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35647.63 MB 2025-02-15 13:14:44,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3427.59 MB 2025-02-15 13:14:44,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42945.48 MB 2025-02-15 13:14:44,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53324.28 MB 2025-02-15 13:14:44,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10378.81 MB 2025-02-15 13:14:44,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51554.20 MB 2025-02-15 13:14:46,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:14:46,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:14:46,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:14:46,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:46,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35647.63 MB 2025-02-15 13:14:46,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36178.47 MB 2025-02-15 13:14:46,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:14:46,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53324.28 MB 2025-02-15 13:14:46,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40911.24 MB 2025-02-15 13:14:46,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12413.04 MB 2025-02-15 13:14:46,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40157.37 MB 2025-02-15 13:14:46,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:14:46,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:14:46,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:14:46,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:46,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36178.47 MB 2025-02-15 13:14:46,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38067.96 MB 2025-02-15 13:14:46,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:14:46,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40911.24 MB 2025-02-15 13:14:46,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42798.68 MB 2025-02-15 13:14:46,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:14:46,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39485.39 MB 2025-02-15 13:14:47,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:14:47,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:14:47,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:14:47,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38067.96 MB 2025-02-15 13:14:47,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40309.82 MB 2025-02-15 13:14:47,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:14:47,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42798.68 MB 2025-02-15 13:14:47,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48460.99 MB 2025-02-15 13:14:47,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:14:47,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45854.10 MB 2025-02-15 13:14:47,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:14:47,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:14:47,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:14:47,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36178.47 MB 2025-02-15 13:14:47,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40309.82 MB 2025-02-15 13:14:47,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:14:47,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40911.24 MB 2025-02-15 13:14:47,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48460.99 MB 2025-02-15 13:14:47,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 13:14:47,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45854.10 MB 2025-02-15 13:14:47,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:14:47,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:14:47,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:14:47,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41017.61 MB 2025-02-15 13:14:47,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41784.61 MB 2025-02-15 13:14:47,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:14:47,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48460.99 MB 2025-02-15 13:14:47,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48876.22 MB 2025-02-15 13:14:47,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:14:47,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42492.40 MB 2025-02-15 13:14:47,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:14:47,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:14:47,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:14:47,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42197.50 MB 2025-02-15 13:14:47,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42403.98 MB 2025-02-15 13:14:47,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.49 MB 2025-02-15 13:14:47,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48876.22 MB 2025-02-15 13:14:47,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48876.22 MB 2025-02-15 13:14:47,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:14:47,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42624.75 MB 2025-02-15 13:14:47,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:14:47,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:14:47,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.73 seconds 2025-02-15 13:14:47,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30822.77 MB 2025-02-15 13:14:47,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42604.74 MB 2025-02-15 13:14:47,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11781.97 MB 2025-02-15 13:14:47,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62987.96 MB 2025-02-15 13:14:47,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48876.22 MB 2025-02-15 13:14:47,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14111.74 MB 2025-02-15 13:14:47,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42624.75 MB 2025-02-15 13:14:47,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:14:47,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:14:47,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:14:47,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42604.74 MB 2025-02-15 13:14:47,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42705.04 MB 2025-02-15 13:14:47,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.31 MB 2025-02-15 13:14:47,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48876.22 MB 2025-02-15 13:14:47,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48876.22 MB 2025-02-15 13:14:47,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:14:47,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43306.89 MB 2025-02-15 13:14:47,485 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 13:14:47,485 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for the video is 2.'] 2025-02-15 13:14:47,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:14:47,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:14:47,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:14:47,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:14:47,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32085.37 MB 2025-02-15 13:14:47,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36273.39 MB 2025-02-15 13:14:47,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.01 MB 2025-02-15 13:14:47,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48876.22 MB 2025-02-15 13:14:47,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59347.30 MB 2025-02-15 13:14:47,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-15 13:14:47,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40461.40 MB 2025-02-15 13:14:47,649 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 13:14:47,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,651 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,652 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:14:47,656 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:14:47,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,657 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:14:47,658 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for the video is 2.'] 2025-02-15 13:14:47,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,658 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,659 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:47,665 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:14:47,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,665 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,666 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:47,666 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:14:47,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,666 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:47,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,667 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:14:47,667 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:14:47,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,667 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:14:47,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,670 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,672 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,672 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:14:47,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:14:47,679 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:15:41,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:15:41,877 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:15:41,882 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:15:41,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:15:41,883 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1425, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:15:41,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:15:41,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1425, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:16:03,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:16:03,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:16:03,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.08 seconds 2025-02-15 13:16:03,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:03,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36781.09 MB 2025-02-15 13:16:03,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41824.08 MB 2025-02-15 13:16:03,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5043.00 MB 2025-02-15 13:16:03,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67008.20 MB 2025-02-15 13:16:03,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:16:03,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16372.47 MB 2025-02-15 13:16:03,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50782.31 MB 2025-02-15 13:16:04,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:16:04,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:16:04,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 13:16:04,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:04,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41824.08 MB 2025-02-15 13:16:04,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37068.73 MB 2025-02-15 13:16:04,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4755.35 MB 2025-02-15 13:16:04,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:16:04,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60597.21 MB 2025-02-15 13:16:04,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9961.47 MB 2025-02-15 13:16:04,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56801.78 MB 2025-02-15 13:16:05,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:16:05,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:16:05,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:16:05,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:05,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37068.73 MB 2025-02-15 13:16:05,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37599.58 MB 2025-02-15 13:16:05,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:16:05,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60597.21 MB 2025-02-15 13:16:05,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:16:05,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9961.47 MB 2025-02-15 13:16:05,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41578.12 MB 2025-02-15 13:16:06,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:16:06,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:16:06,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:16:06,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37599.58 MB 2025-02-15 13:16:06,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39489.04 MB 2025-02-15 13:16:06,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.46 MB 2025-02-15 13:16:06,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:16:06,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:16:06,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:16:06,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40906.47 MB 2025-02-15 13:16:06,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:16:06,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:16:06,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:16:06,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39489.04 MB 2025-02-15 13:16:06,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41730.89 MB 2025-02-15 13:16:06,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:16:06,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:16:06,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:16:06,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:16:06,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47275.17 MB 2025-02-15 13:16:06,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:16:06,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:16:06,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:16:06,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37599.58 MB 2025-02-15 13:16:06,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41730.89 MB 2025-02-15 13:16:06,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-15 13:16:06,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:16:06,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:16:06,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:16:06,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47275.17 MB 2025-02-15 13:16:06,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:16:06,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:16:06,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:16:06,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42438.68 MB 2025-02-15 13:16:06,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43205.68 MB 2025-02-15 13:16:06,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:16:06,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:16:06,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51050.97 MB 2025-02-15 13:16:06,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:16:06,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43913.47 MB 2025-02-15 13:16:06,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:16:06,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:16:06,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:16:06,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43618.57 MB 2025-02-15 13:16:06,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43825.53 MB 2025-02-15 13:16:06,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.96 MB 2025-02-15 13:16:06,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51050.97 MB 2025-02-15 13:16:06,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51050.97 MB 2025-02-15 13:16:06,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:16:06,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44048.16 MB 2025-02-15 13:16:06,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:16:06,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:16:06,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.51 seconds 2025-02-15 13:16:06,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31816.27 MB 2025-02-15 13:16:06,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44026.61 MB 2025-02-15 13:16:06,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12210.33 MB 2025-02-15 13:16:06,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67008.20 MB 2025-02-15 13:16:06,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51050.97 MB 2025-02-15 13:16:06,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15957.23 MB 2025-02-15 13:16:06,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44048.16 MB 2025-02-15 13:16:06,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:16:06,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:16:06,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:16:06,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44026.61 MB 2025-02-15 13:16:06,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44127.07 MB 2025-02-15 13:16:06,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:16:06,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51050.97 MB 2025-02-15 13:16:06,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51050.97 MB 2025-02-15 13:16:06,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:16:06,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44729.87 MB 2025-02-15 13:16:06,685 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:16:06,685 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:16:06,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:16:06,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:16:06,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:16:06,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:16:06,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33079.19 MB 2025-02-15 13:16:06,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37273.68 MB 2025-02-15 13:16:06,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:16:06,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51050.97 MB 2025-02-15 13:16:06,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55245.28 MB 2025-02-15 13:16:06,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:16:06,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41467.98 MB 2025-02-15 13:16:06,852 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:16:06,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,853 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,854 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:16:06,859 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:16:06,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,860 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:16:06,860 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:16:06,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,861 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,861 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:16:06,867 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:16:06,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,868 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,868 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:16:06,868 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:16:06,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,869 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:16:06,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,869 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:16:06,869 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:16:06,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,870 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:16:06,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,873 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,874 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,875 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:16:06,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:16:06,882 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:00,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:00,057 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:00,064 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:17:00,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:00,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1245, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:17:00,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:00,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1245, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:17:19,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:17:19,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:17:19,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.36 seconds 2025-02-15 13:17:19,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:19,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35647.88 MB 2025-02-15 13:17:19,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40054.00 MB 2025-02-15 13:17:19,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4406.12 MB 2025-02-15 13:17:19,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63027.81 MB 2025-02-15 13:17:19,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51818.53 MB 2025-02-15 13:17:19,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11209.28 MB 2025-02-15 13:17:19,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48969.63 MB 2025-02-15 13:17:19,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:17:19,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:17:19,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:17:19,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:19,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40054.00 MB 2025-02-15 13:17:19,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36254.04 MB 2025-02-15 13:17:19,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3799.96 MB 2025-02-15 13:17:19,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51818.53 MB 2025-02-15 13:17:19,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60563.65 MB 2025-02-15 13:17:19,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8745.12 MB 2025-02-15 13:17:19,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53204.53 MB 2025-02-15 13:17:21,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:17:21,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:17:21,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:17:21,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36254.04 MB 2025-02-15 13:17:21,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36784.88 MB 2025-02-15 13:17:21,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:17:21,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60563.65 MB 2025-02-15 13:17:21,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47412.41 MB 2025-02-15 13:17:21,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13151.24 MB 2025-02-15 13:17:21,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40763.42 MB 2025-02-15 13:17:21,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:17:21,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:17:21,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:17:21,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36784.88 MB 2025-02-15 13:17:21,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38674.37 MB 2025-02-15 13:17:21,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:17:21,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47412.41 MB 2025-02-15 13:17:21,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47412.41 MB 2025-02-15 13:17:21,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:17:21,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40091.80 MB 2025-02-15 13:17:21,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:17:21,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:17:21,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:17:21,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38674.37 MB 2025-02-15 13:17:21,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40916.23 MB 2025-02-15 13:17:21,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:17:21,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47412.41 MB 2025-02-15 13:17:21,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49771.71 MB 2025-02-15 13:17:21,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 13:17:21,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46460.51 MB 2025-02-15 13:17:21,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:17:21,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:17:21,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:17:21,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36784.88 MB 2025-02-15 13:17:21,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40916.23 MB 2025-02-15 13:17:21,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:17:21,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47412.41 MB 2025-02-15 13:17:21,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49771.71 MB 2025-02-15 13:17:21,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 13:17:21,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46460.51 MB 2025-02-15 13:17:21,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:17:21,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:17:21,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:17:21,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41624.02 MB 2025-02-15 13:17:21,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42391.02 MB 2025-02-15 13:17:21,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:17:21,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49771.71 MB 2025-02-15 13:17:21,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50186.94 MB 2025-02-15 13:17:21,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:17:21,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43098.81 MB 2025-02-15 13:17:21,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:17:21,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:17:21,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:17:21,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42803.91 MB 2025-02-15 13:17:21,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43009.47 MB 2025-02-15 13:17:21,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.56 MB 2025-02-15 13:17:21,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50186.94 MB 2025-02-15 13:17:21,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50186.94 MB 2025-02-15 13:17:21,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:17:21,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43233.87 MB 2025-02-15 13:17:21,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:17:21,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:17:21,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.78 seconds 2025-02-15 13:17:21,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:21,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31310.20 MB 2025-02-15 13:17:21,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43209.73 MB 2025-02-15 13:17:21,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11899.53 MB 2025-02-15 13:17:21,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63027.81 MB 2025-02-15 13:17:21,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50186.94 MB 2025-02-15 13:17:21,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12840.86 MB 2025-02-15 13:17:21,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43233.87 MB 2025-02-15 13:17:22,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:17:22,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:17:22,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:17:22,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:22,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43209.73 MB 2025-02-15 13:17:22,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43309.79 MB 2025-02-15 13:17:22,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.06 MB 2025-02-15 13:17:22,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50186.94 MB 2025-02-15 13:17:22,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50186.94 MB 2025-02-15 13:17:22,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:17:22,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43910.16 MB 2025-02-15 13:17:22,137 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 13:17:22,137 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:17:22,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:17:22,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:17:22,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:17:22,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:17:22,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32572.31 MB 2025-02-15 13:17:22,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36749.87 MB 2025-02-15 13:17:22,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.56 MB 2025-02-15 13:17:22,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50186.94 MB 2025-02-15 13:17:22,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54364.47 MB 2025-02-15 13:17:22,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 13:17:22,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40927.40 MB 2025-02-15 13:17:22,301 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 13:17:22,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:17:22,308 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:17:22,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,309 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:17:22,309 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:17:22,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,310 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,311 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:22,316 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:17:22,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,317 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,317 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:22,317 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:17:22,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,318 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:22,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,318 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:17:22,318 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:17:22,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,319 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:17:22,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,322 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,323 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,324 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:17:22,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:17:22,330 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:19,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:19,533 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:19,538 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:18:19,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:19,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1187, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:18:19,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:19,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1187, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:18:37,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:18:37,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:18:37,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.42 seconds 2025-02-15 13:18:37,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:37,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35365.95 MB 2025-02-15 13:18:37,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39566.68 MB 2025-02-15 13:18:37,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4200.73 MB 2025-02-15 13:18:37,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62268.64 MB 2025-02-15 13:18:37,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43375.39 MB 2025-02-15 13:18:37,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18893.24 MB 2025-02-15 13:18:37,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48462.01 MB 2025-02-15 13:18:38,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:18:38,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:18:38,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 13:18:38,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:38,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39566.68 MB 2025-02-15 13:18:38,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36075.78 MB 2025-02-15 13:18:38,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3490.90 MB 2025-02-15 13:18:38,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43375.39 MB 2025-02-15 13:18:38,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53823.41 MB 2025-02-15 13:18:38,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10448.01 MB 2025-02-15 13:18:38,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52116.11 MB 2025-02-15 13:18:40,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:18:40,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:18:40,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:18:40,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36075.78 MB 2025-02-15 13:18:40,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36606.62 MB 2025-02-15 13:18:40,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:18:40,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53823.41 MB 2025-02-15 13:18:40,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41297.12 MB 2025-02-15 13:18:40,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12526.29 MB 2025-02-15 13:18:40,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40585.17 MB 2025-02-15 13:18:40,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:18:40,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:18:40,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:18:40,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36606.62 MB 2025-02-15 13:18:40,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38496.09 MB 2025-02-15 13:18:40,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-15 13:18:40,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41297.12 MB 2025-02-15 13:18:40,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43184.55 MB 2025-02-15 13:18:40,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:18:40,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39913.52 MB 2025-02-15 13:18:40,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:18:40,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:18:40,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:18:40,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38496.09 MB 2025-02-15 13:18:40,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40737.95 MB 2025-02-15 13:18:40,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:18:40,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43184.55 MB 2025-02-15 13:18:40,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48846.86 MB 2025-02-15 13:18:40,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:18:40,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46282.23 MB 2025-02-15 13:18:40,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:18:40,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:18:40,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:18:40,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36606.62 MB 2025-02-15 13:18:40,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40737.95 MB 2025-02-15 13:18:40,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-15 13:18:40,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41297.12 MB 2025-02-15 13:18:40,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48846.86 MB 2025-02-15 13:18:40,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 13:18:40,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46282.23 MB 2025-02-15 13:18:40,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:18:40,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:18:40,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:18:40,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41445.74 MB 2025-02-15 13:18:40,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42212.74 MB 2025-02-15 13:18:40,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:18:40,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48846.86 MB 2025-02-15 13:18:40,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49262.10 MB 2025-02-15 13:18:40,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:18:40,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42920.53 MB 2025-02-15 13:18:40,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:18:40,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:18:40,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:18:40,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42625.63 MB 2025-02-15 13:18:40,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42832.11 MB 2025-02-15 13:18:40,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.48 MB 2025-02-15 13:18:40,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49262.10 MB 2025-02-15 13:18:40,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49262.10 MB 2025-02-15 13:18:40,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:18:40,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43054.24 MB 2025-02-15 13:18:40,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:18:40,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:18:40,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.89 seconds 2025-02-15 13:18:40,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31230.35 MB 2025-02-15 13:18:40,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43033.01 MB 2025-02-15 13:18:40,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11802.66 MB 2025-02-15 13:18:40,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62268.64 MB 2025-02-15 13:18:40,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49262.10 MB 2025-02-15 13:18:40,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13006.54 MB 2025-02-15 13:18:40,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43054.24 MB 2025-02-15 13:18:40,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:18:40,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:18:40,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:18:40,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43033.01 MB 2025-02-15 13:18:40,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43133.39 MB 2025-02-15 13:18:40,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.38 MB 2025-02-15 13:18:40,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49262.10 MB 2025-02-15 13:18:40,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49262.10 MB 2025-02-15 13:18:40,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:18:40,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43735.67 MB 2025-02-15 13:18:40,717 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 13:18:40,718 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:18:40,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:18:40,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:18:40,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:18:40,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:18:40,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32493.10 MB 2025-02-15 13:18:40,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36683.99 MB 2025-02-15 13:18:40,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.89 MB 2025-02-15 13:18:40,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49262.10 MB 2025-02-15 13:18:40,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57646.51 MB 2025-02-15 13:18:40,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 13:18:40,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40874.37 MB 2025-02-15 13:18:40,886 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 13:18:40,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,888 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,888 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:18:40,893 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:18:40,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,894 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:18:40,894 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:18:40,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,895 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,896 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:40,901 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:18:40,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,902 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,903 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:40,903 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:18:40,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,903 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:40,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,904 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:18:40,904 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:18:40,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,904 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:18:40,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,910 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,911 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,912 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:18:40,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:18:40,920 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:08,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:08,106 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:08,110 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:19:08,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:08,112 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:19:08,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:08,113 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:19:27,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:19:27,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:19:27,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.58 seconds 2025-02-15 13:19:27,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:27,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36029.72 MB 2025-02-15 13:19:27,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40506.48 MB 2025-02-15 13:19:27,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4476.76 MB 2025-02-15 13:19:27,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65672.31 MB 2025-02-15 13:19:27,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51390.71 MB 2025-02-15 13:19:27,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14281.61 MB 2025-02-15 13:19:27,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49351.46 MB 2025-02-15 13:19:27,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:19:27,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:19:27,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:19:27,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:27,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40506.48 MB 2025-02-15 13:19:27,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36600.48 MB 2025-02-15 13:19:27,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.00 MB 2025-02-15 13:19:27,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51390.71 MB 2025-02-15 13:19:27,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60244.89 MB 2025-02-15 13:19:27,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8854.18 MB 2025-02-15 13:19:27,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53816.14 MB 2025-02-15 13:19:29,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:19:29,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:19:29,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:19:29,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:29,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36600.48 MB 2025-02-15 13:19:29,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37131.32 MB 2025-02-15 13:19:29,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:19:29,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60244.89 MB 2025-02-15 13:19:29,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 13:19:29,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13031.70 MB 2025-02-15 13:19:29,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41110.64 MB 2025-02-15 13:19:29,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:19:29,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:19:29,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:19:29,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:29,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37131.32 MB 2025-02-15 13:19:29,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39020.81 MB 2025-02-15 13:19:29,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:19:29,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 13:19:29,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 13:19:29,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:19:29,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40438.24 MB 2025-02-15 13:19:29,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:19:29,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:19:29,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:19:29,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:29,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39020.81 MB 2025-02-15 13:19:29,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41262.67 MB 2025-02-15 13:19:29,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:19:29,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 13:19:29,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50044.34 MB 2025-02-15 13:19:29,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 13:19:29,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46806.95 MB 2025-02-15 13:19:29,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:19:29,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:19:29,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:19:29,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:29,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37131.32 MB 2025-02-15 13:19:29,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41262.67 MB 2025-02-15 13:19:29,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:19:29,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 13:19:29,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50044.34 MB 2025-02-15 13:19:29,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 13:19:29,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46806.95 MB 2025-02-15 13:19:30,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:19:30,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:19:30,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 13:19:30,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:30,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41970.46 MB 2025-02-15 13:19:30,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42737.46 MB 2025-02-15 13:19:30,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:19:30,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50044.34 MB 2025-02-15 13:19:30,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50459.57 MB 2025-02-15 13:19:30,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:19:30,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43445.25 MB 2025-02-15 13:19:30,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:19:30,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:19:30,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:19:30,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:30,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43150.35 MB 2025-02-15 13:19:30,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43356.99 MB 2025-02-15 13:19:30,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.65 MB 2025-02-15 13:19:30,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50459.57 MB 2025-02-15 13:19:30,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50459.57 MB 2025-02-15 13:19:30,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:19:30,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43577.58 MB 2025-02-15 13:19:30,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:19:30,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:19:30,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.99 seconds 2025-02-15 13:19:30,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:30,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31622.35 MB 2025-02-15 13:19:30,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43557.72 MB 2025-02-15 13:19:30,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11935.37 MB 2025-02-15 13:19:30,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65672.31 MB 2025-02-15 13:19:30,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50459.57 MB 2025-02-15 13:19:30,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15212.74 MB 2025-02-15 13:19:30,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43577.58 MB 2025-02-15 13:19:30,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:19:30,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:19:30,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:19:30,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:30,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43557.72 MB 2025-02-15 13:19:30,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43658.02 MB 2025-02-15 13:19:30,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.29 MB 2025-02-15 13:19:30,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50459.57 MB 2025-02-15 13:19:30,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50459.57 MB 2025-02-15 13:19:30,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:19:30,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44259.79 MB 2025-02-15 13:19:30,386 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 13:19:30,386 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:19:30,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:19:30,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:19:30,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:19:30,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:19:30,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32884.93 MB 2025-02-15 13:19:30,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37072.23 MB 2025-02-15 13:19:30,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4187.30 MB 2025-02-15 13:19:30,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50459.57 MB 2025-02-15 13:19:30,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50459.57 MB 2025-02-15 13:19:30,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:19:30,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41259.02 MB 2025-02-15 13:19:30,555 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 13:19:30,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,557 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,558 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:19:30,563 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:19:30,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,564 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:19:30,564 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:19:30,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,565 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,565 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:30,571 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:19:30,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,572 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,572 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:30,572 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:19:30,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,573 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:30,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,573 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:19:30,573 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:19:30,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,574 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:19:30,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,577 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,578 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,579 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:19:30,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:19:30,586 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:27,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:27,539 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:27,544 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:20:27,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:27,546 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 755, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:20:27,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:27,546 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 755, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:20:39,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:20:39,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:20:39,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.63 seconds 2025-02-15 13:20:39,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:39,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32597.87 MB 2025-02-15 13:20:39,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35269.77 MB 2025-02-15 13:20:39,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2671.90 MB 2025-02-15 13:20:39,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58607.01 MB 2025-02-15 13:20:39,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43146.81 MB 2025-02-15 13:20:39,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15460.20 MB 2025-02-15 13:20:39,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44107.67 MB 2025-02-15 13:20:39,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:20:39,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:20:39,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 13:20:39,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:39,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35269.77 MB 2025-02-15 13:20:39,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34072.11 MB 2025-02-15 13:20:39,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1197.66 MB 2025-02-15 13:20:39,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43146.81 MB 2025-02-15 13:20:39,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-15 13:20:39,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4490.00 MB 2025-02-15 13:20:39,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44186.98 MB 2025-02-15 13:20:41,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:20:41,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:20:41,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 13:20:41,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34072.11 MB 2025-02-15 13:20:41,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34602.95 MB 2025-02-15 13:20:41,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:20:41,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-15 13:20:41,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45271.22 MB 2025-02-15 13:20:41,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2365.59 MB 2025-02-15 13:20:41,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38581.50 MB 2025-02-15 13:20:41,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:20:41,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:20:41,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:20:41,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34602.95 MB 2025-02-15 13:20:41,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36492.29 MB 2025-02-15 13:20:41,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.34 MB 2025-02-15 13:20:41,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45271.22 MB 2025-02-15 13:20:41,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45271.22 MB 2025-02-15 13:20:41,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:20:41,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37909.85 MB 2025-02-15 13:20:41,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:20:41,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:20:41,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:20:41,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36492.29 MB 2025-02-15 13:20:41,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38734.28 MB 2025-02-15 13:20:41,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.99 MB 2025-02-15 13:20:41,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45271.22 MB 2025-02-15 13:20:41,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47158.66 MB 2025-02-15 13:20:41,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:20:41,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44278.56 MB 2025-02-15 13:20:41,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:20:41,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:20:41,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:20:41,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34602.95 MB 2025-02-15 13:20:41,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38734.28 MB 2025-02-15 13:20:41,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.33 MB 2025-02-15 13:20:41,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45271.22 MB 2025-02-15 13:20:41,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47158.66 MB 2025-02-15 13:20:41,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:20:41,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44278.56 MB 2025-02-15 13:20:41,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:20:41,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:20:41,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:20:41,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39442.06 MB 2025-02-15 13:20:41,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40209.07 MB 2025-02-15 13:20:41,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:20:41,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47158.66 MB 2025-02-15 13:20:41,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47573.89 MB 2025-02-15 13:20:41,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:20:41,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40916.86 MB 2025-02-15 13:20:41,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:20:41,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:20:41,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:20:41,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40621.96 MB 2025-02-15 13:20:41,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40829.24 MB 2025-02-15 13:20:41,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.28 MB 2025-02-15 13:20:41,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47573.89 MB 2025-02-15 13:20:41,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47573.89 MB 2025-02-15 13:20:41,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:20:41,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41029.23 MB 2025-02-15 13:20:41,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:20:41,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:20:41,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.04 seconds 2025-02-15 13:20:41,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29967.38 MB 2025-02-15 13:20:41,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41030.31 MB 2025-02-15 13:20:41,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11062.92 MB 2025-02-15 13:20:41,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58607.01 MB 2025-02-15 13:20:41,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47573.89 MB 2025-02-15 13:20:41,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11033.12 MB 2025-02-15 13:20:41,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41030.31 MB 2025-02-15 13:20:41,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:20:41,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:20:41,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:20:41,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41030.31 MB 2025-02-15 13:20:41,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41130.78 MB 2025-02-15 13:20:41,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:20:41,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47573.89 MB 2025-02-15 13:20:41,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47573.89 MB 2025-02-15 13:20:41,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:20:41,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41733.58 MB 2025-02-15 13:20:41,874 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:20:41,874 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 13:20:41,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:20:41,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:20:41,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:20:41,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:20:41,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31230.31 MB 2025-02-15 13:20:41,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35424.79 MB 2025-02-15 13:20:41,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:20:41,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47573.89 MB 2025-02-15 13:20:41,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51768.20 MB 2025-02-15 13:20:41,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:20:41,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39619.10 MB 2025-02-15 13:20:42,047 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:20:42,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,048 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,049 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:20:42,054 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:20:42,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,055 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:20:42,055 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 13:20:42,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,056 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,057 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:42,064 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:20:42,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,064 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,065 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:42,065 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:20:42,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,065 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:42,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,066 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:20:42,066 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:20:42,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,066 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:20:42,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,070 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,070 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,071 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:20:42,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:20:42,078 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:21:41,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:21:41,086 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:21:41,095 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:21:41,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:21:41,097 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1215, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:21:41,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:21:41,099 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1215, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:21:59,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:21:59,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:21:59,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.74 seconds 2025-02-15 13:21:59,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:21:59,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35924.76 MB 2025-02-15 13:21:59,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40224.58 MB 2025-02-15 13:21:59,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4299.82 MB 2025-02-15 13:21:59,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60037.27 MB 2025-02-15 13:21:59,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51764.00 MB 2025-02-15 13:21:59,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8273.26 MB 2025-02-15 13:21:59,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49102.74 MB 2025-02-15 13:21:59,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:21:59,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:21:59,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:21:59,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:21:59,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40224.58 MB 2025-02-15 13:21:59,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36584.00 MB 2025-02-15 13:21:59,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3640.58 MB 2025-02-15 13:21:59,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51764.00 MB 2025-02-15 13:21:59,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58122.57 MB 2025-02-15 13:21:59,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6358.56 MB 2025-02-15 13:21:59,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53004.86 MB 2025-02-15 13:22:01,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:22:01,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:22:01,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 13:22:01,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:01,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36584.00 MB 2025-02-15 13:22:01,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37114.84 MB 2025-02-15 13:22:01,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:22:01,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58122.57 MB 2025-02-15 13:22:01,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47462.74 MB 2025-02-15 13:22:01,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10659.82 MB 2025-02-15 13:22:01,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41093.39 MB 2025-02-15 13:22:01,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:22:01,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:22:01,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:22:01,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:01,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37114.84 MB 2025-02-15 13:22:01,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39004.33 MB 2025-02-15 13:22:01,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:22:01,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47462.74 MB 2025-02-15 13:22:01,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47462.74 MB 2025-02-15 13:22:01,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:22:01,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40421.76 MB 2025-02-15 13:22:02,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:22:02,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:22:02,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:22:02,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39004.33 MB 2025-02-15 13:22:02,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41246.19 MB 2025-02-15 13:22:02,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:22:02,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47462.74 MB 2025-02-15 13:22:02,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50293.90 MB 2025-02-15 13:22:02,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 13:22:02,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46790.47 MB 2025-02-15 13:22:02,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:22:02,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:22:02,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:22:02,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37114.84 MB 2025-02-15 13:22:02,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41246.19 MB 2025-02-15 13:22:02,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:22:02,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47462.74 MB 2025-02-15 13:22:02,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50293.90 MB 2025-02-15 13:22:02,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 13:22:02,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46790.47 MB 2025-02-15 13:22:02,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:22:02,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:22:02,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:22:02,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41953.98 MB 2025-02-15 13:22:02,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42720.98 MB 2025-02-15 13:22:02,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:22:02,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50293.90 MB 2025-02-15 13:22:02,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50709.14 MB 2025-02-15 13:22:02,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:22:02,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43428.77 MB 2025-02-15 13:22:02,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:22:02,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:22:02,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:22:02,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43133.87 MB 2025-02-15 13:22:02,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43339.48 MB 2025-02-15 13:22:02,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.61 MB 2025-02-15 13:22:02,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50709.14 MB 2025-02-15 13:22:02,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50709.14 MB 2025-02-15 13:22:02,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:22:02,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43562.16 MB 2025-02-15 13:22:02,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:22:02,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:22:02,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.14 seconds 2025-02-15 13:22:02,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31691.60 MB 2025-02-15 13:22:02,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43539.64 MB 2025-02-15 13:22:02,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11848.04 MB 2025-02-15 13:22:02,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60037.27 MB 2025-02-15 13:22:02,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50709.14 MB 2025-02-15 13:22:02,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9328.13 MB 2025-02-15 13:22:02,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43562.16 MB 2025-02-15 13:22:02,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:22:02,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:22:02,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:22:02,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43539.64 MB 2025-02-15 13:22:02,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43639.66 MB 2025-02-15 13:22:02,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.01 MB 2025-02-15 13:22:02,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50709.14 MB 2025-02-15 13:22:02,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50709.14 MB 2025-02-15 13:22:02,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:22:02,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44239.73 MB 2025-02-15 13:22:02,526 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 13:22:02,526 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for the video is 2.'] 2025-02-15 13:22:02,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:22:02,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:22:02,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:22:02,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:22:02,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32953.61 MB 2025-02-15 13:22:02,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37129.12 MB 2025-02-15 13:22:02,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.50 MB 2025-02-15 13:22:02,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50709.14 MB 2025-02-15 13:22:02,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54884.56 MB 2025-02-15 13:22:02,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 13:22:02,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41304.55 MB 2025-02-15 13:22:02,689 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 13:22:02,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,691 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,692 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:22:02,696 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:22:02,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,697 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:22:02,697 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for the video is 2.'] 2025-02-15 13:22:02,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,698 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,699 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:22:02,705 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:22:02,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,705 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,706 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:22:02,706 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:22:02,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,706 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:22:02,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,706 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:22:02,707 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:22:02,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,707 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:22:02,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,710 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,711 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,712 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:22:02,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:22:02,719 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:02,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:02,331 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:02,337 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:23:02,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:02,338 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1122, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:23:02,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:02,339 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1122, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:23:19,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:23:19,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:23:19,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.35 seconds 2025-02-15 13:23:19,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:19,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35398.45 MB 2025-02-15 13:23:19,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39369.14 MB 2025-02-15 13:23:19,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3970.70 MB 2025-02-15 13:23:19,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63275.27 MB 2025-02-15 13:23:19,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43633.34 MB 2025-02-15 13:23:19,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19641.93 MB 2025-02-15 13:23:19,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48268.01 MB 2025-02-15 13:23:19,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:23:19,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:23:19,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 13:23:19,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:19,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39369.14 MB 2025-02-15 13:23:19,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36223.29 MB 2025-02-15 13:23:19,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3145.85 MB 2025-02-15 13:23:19,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43633.34 MB 2025-02-15 13:23:19,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53632.57 MB 2025-02-15 13:23:19,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9999.22 MB 2025-02-15 13:23:19,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51469.78 MB 2025-02-15 13:23:21,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:23:21,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:23:21,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:23:21,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:21,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36223.29 MB 2025-02-15 13:23:21,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36754.13 MB 2025-02-15 13:23:21,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:23:21,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53632.57 MB 2025-02-15 13:23:21,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41785.75 MB 2025-02-15 13:23:21,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11846.81 MB 2025-02-15 13:23:21,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40732.68 MB 2025-02-15 13:23:21,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:23:21,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:23:21,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:23:21,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:21,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36754.13 MB 2025-02-15 13:23:21,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38643.34 MB 2025-02-15 13:23:21,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-15 13:23:21,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41785.75 MB 2025-02-15 13:23:21,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43673.19 MB 2025-02-15 13:23:21,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:23:21,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40060.77 MB 2025-02-15 13:23:21,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:23:21,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:23:21,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:23:21,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:21,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38643.34 MB 2025-02-15 13:23:21,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40885.20 MB 2025-02-15 13:23:21,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:23:21,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43673.19 MB 2025-02-15 13:23:21,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49335.50 MB 2025-02-15 13:23:21,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:23:21,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46429.48 MB 2025-02-15 13:23:21,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:23:21,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:23:21,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:23:21,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:21,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36754.13 MB 2025-02-15 13:23:21,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40885.20 MB 2025-02-15 13:23:21,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-15 13:23:21,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41785.75 MB 2025-02-15 13:23:21,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49335.50 MB 2025-02-15 13:23:21,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 13:23:21,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46429.48 MB 2025-02-15 13:23:22,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:23:22,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:23:22,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:23:22,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:22,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41592.99 MB 2025-02-15 13:23:22,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42359.99 MB 2025-02-15 13:23:22,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:23:22,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49335.50 MB 2025-02-15 13:23:22,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49750.74 MB 2025-02-15 13:23:22,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:23:22,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43067.78 MB 2025-02-15 13:23:22,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:23:22,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:23:22,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:23:22,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:22,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42772.88 MB 2025-02-15 13:23:22,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42978.31 MB 2025-02-15 13:23:22,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.44 MB 2025-02-15 13:23:22,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49750.74 MB 2025-02-15 13:23:22,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49750.74 MB 2025-02-15 13:23:22,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:23:22,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43199.15 MB 2025-02-15 13:23:22,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:23:22,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:23:22,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.78 seconds 2025-02-15 13:23:22,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:22,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31489.31 MB 2025-02-15 13:23:22,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43178.23 MB 2025-02-15 13:23:22,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11688.92 MB 2025-02-15 13:23:22,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63275.27 MB 2025-02-15 13:23:22,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49750.74 MB 2025-02-15 13:23:22,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13524.53 MB 2025-02-15 13:23:22,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43199.15 MB 2025-02-15 13:23:22,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:23:22,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:23:22,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:23:22,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:22,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43178.23 MB 2025-02-15 13:23:22,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43278.12 MB 2025-02-15 13:23:22,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.89 MB 2025-02-15 13:23:22,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49750.74 MB 2025-02-15 13:23:22,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49750.74 MB 2025-02-15 13:23:22,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:23:22,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43877.45 MB 2025-02-15 13:23:22,399 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 13:23:22,400 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:23:22,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:23:22,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:23:22,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:23:22,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:23:22,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32751.07 MB 2025-02-15 13:23:22,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36922.31 MB 2025-02-15 13:23:22,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4171.24 MB 2025-02-15 13:23:22,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49750.74 MB 2025-02-15 13:23:22,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60179.87 MB 2025-02-15 13:23:22,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10429.14 MB 2025-02-15 13:23:22,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41092.17 MB 2025-02-15 13:23:22,564 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 13:23:22,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,565 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,566 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:23:22,571 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:23:22,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,572 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:23:22,572 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:23:22,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,573 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,573 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:22,579 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:23:22,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,580 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,580 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:22,580 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:23:22,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,581 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:22,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,581 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:23:22,581 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:23:22,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,582 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:23:22,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,587 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,589 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,591 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:23:22,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:23:22,598 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:15,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:15,060 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:15,065 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:24:15,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:15,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1216, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:24:15,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:15,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1216, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:24:33,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:24:33,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:24:33,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.80 seconds 2025-02-15 13:24:33,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:33,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36175.18 MB 2025-02-15 13:24:33,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40478.54 MB 2025-02-15 13:24:33,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4303.36 MB 2025-02-15 13:24:33,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68692.21 MB 2025-02-15 13:24:33,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50323.26 MB 2025-02-15 13:24:33,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18368.95 MB 2025-02-15 13:24:33,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49496.92 MB 2025-02-15 13:24:33,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:24:33,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:24:33,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:24:33,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:33,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40478.54 MB 2025-02-15 13:24:33,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36832.65 MB 2025-02-15 13:24:33,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3645.89 MB 2025-02-15 13:24:33,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50323.26 MB 2025-02-15 13:24:33,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58753.81 MB 2025-02-15 13:24:33,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8430.55 MB 2025-02-15 13:24:33,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53154.48 MB 2025-02-15 13:24:35,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:24:35,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:24:35,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:24:35,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:35,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36832.65 MB 2025-02-15 13:24:35,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37363.49 MB 2025-02-15 13:24:35,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:24:35,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58753.81 MB 2025-02-15 13:24:35,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46019.90 MB 2025-02-15 13:24:35,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12733.91 MB 2025-02-15 13:24:35,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41342.03 MB 2025-02-15 13:24:35,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:24:35,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:24:35,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:24:35,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:35,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37363.49 MB 2025-02-15 13:24:35,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39252.63 MB 2025-02-15 13:24:35,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.14 MB 2025-02-15 13:24:35,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46019.90 MB 2025-02-15 13:24:35,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46019.90 MB 2025-02-15 13:24:35,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:24:35,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40670.06 MB 2025-02-15 13:24:36,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:24:36,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:24:36,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:24:36,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39252.63 MB 2025-02-15 13:24:36,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41494.48 MB 2025-02-15 13:24:36,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:24:36,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46019.90 MB 2025-02-15 13:24:36,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50266.64 MB 2025-02-15 13:24:36,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 13:24:36,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47038.77 MB 2025-02-15 13:24:36,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:24:36,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:24:36,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:24:36,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37363.49 MB 2025-02-15 13:24:36,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41494.48 MB 2025-02-15 13:24:36,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.00 MB 2025-02-15 13:24:36,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46019.90 MB 2025-02-15 13:24:36,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50266.64 MB 2025-02-15 13:24:36,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 13:24:36,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47038.77 MB 2025-02-15 13:24:36,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:24:36,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:24:36,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:24:36,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42202.27 MB 2025-02-15 13:24:36,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42969.27 MB 2025-02-15 13:24:36,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:24:36,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50266.64 MB 2025-02-15 13:24:36,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50681.87 MB 2025-02-15 13:24:36,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:24:36,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43677.06 MB 2025-02-15 13:24:36,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:24:36,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:24:36,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:24:36,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43382.16 MB 2025-02-15 13:24:36,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43587.73 MB 2025-02-15 13:24:36,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.56 MB 2025-02-15 13:24:36,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50681.87 MB 2025-02-15 13:24:36,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50681.87 MB 2025-02-15 13:24:36,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:24:36,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43811.62 MB 2025-02-15 13:24:36,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:24:36,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:24:36,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.21 seconds 2025-02-15 13:24:36,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31938.54 MB 2025-02-15 13:24:36,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43788.63 MB 2025-02-15 13:24:36,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11850.09 MB 2025-02-15 13:24:36,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68692.21 MB 2025-02-15 13:24:36,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50681.87 MB 2025-02-15 13:24:36,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18010.34 MB 2025-02-15 13:24:36,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43811.62 MB 2025-02-15 13:24:36,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:24:36,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:24:36,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:24:36,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43788.63 MB 2025-02-15 13:24:36,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43889.01 MB 2025-02-15 13:24:36,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.38 MB 2025-02-15 13:24:36,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50681.87 MB 2025-02-15 13:24:36,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50681.87 MB 2025-02-15 13:24:36,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:24:36,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44491.29 MB 2025-02-15 13:24:36,560 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 13:24:36,560 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:24:36,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:24:36,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:24:36,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:24:36,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:24:36,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33201.29 MB 2025-02-15 13:24:36,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37392.18 MB 2025-02-15 13:24:36,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.89 MB 2025-02-15 13:24:36,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50681.87 MB 2025-02-15 13:24:36,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59066.29 MB 2025-02-15 13:24:36,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 13:24:36,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41582.56 MB 2025-02-15 13:24:36,727 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 13:24:36,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,728 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,729 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:24:36,734 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:24:36,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,735 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:24:36,735 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:24:36,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,736 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,737 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:36,742 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:24:36,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,743 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,743 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:36,743 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:24:36,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,744 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:36,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,744 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:24:36,744 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:24:36,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,745 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:24:36,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,748 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,749 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,750 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:24:36,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:24:36,758 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:25:54,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:25:54,904 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:25:54,909 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:25:54,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:25:54,910 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1095, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:25:54,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:25:54,911 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1095, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:26:11,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:26:11,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:26:11,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.89 seconds 2025-02-15 13:26:11,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:11,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35453.76 MB 2025-02-15 13:26:11,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39329.30 MB 2025-02-15 13:26:11,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3875.54 MB 2025-02-15 13:26:11,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67700.26 MB 2025-02-15 13:26:11,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45845.84 MB 2025-02-15 13:26:11,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21854.42 MB 2025-02-15 13:26:11,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48323.32 MB 2025-02-15 13:26:11,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:26:11,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:26:11,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:26:11,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:11,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39329.30 MB 2025-02-15 13:26:11,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36325.33 MB 2025-02-15 13:26:11,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3003.96 MB 2025-02-15 13:26:11,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45845.84 MB 2025-02-15 13:26:11,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54708.40 MB 2025-02-15 13:26:11,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8862.56 MB 2025-02-15 13:26:11,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50999.76 MB 2025-02-15 13:26:13,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:26:13,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:26:13,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 13:26:13,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:13,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36325.33 MB 2025-02-15 13:26:13,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36856.17 MB 2025-02-15 13:26:13,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:26:13,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54708.40 MB 2025-02-15 13:26:13,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41299.21 MB 2025-02-15 13:26:13,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13409.19 MB 2025-02-15 13:26:13,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40835.76 MB 2025-02-15 13:26:13,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:26:13,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:26:13,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:26:13,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:13,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36856.17 MB 2025-02-15 13:26:13,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38745.67 MB 2025-02-15 13:26:13,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:26:13,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41299.21 MB 2025-02-15 13:26:13,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43186.65 MB 2025-02-15 13:26:13,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:26:13,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40163.09 MB 2025-02-15 13:26:14,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:26:14,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:26:14,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:26:14,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38745.67 MB 2025-02-15 13:26:14,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40987.52 MB 2025-02-15 13:26:14,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:26:14,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43186.65 MB 2025-02-15 13:26:14,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49320.82 MB 2025-02-15 13:26:14,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:26:14,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46531.80 MB 2025-02-15 13:26:14,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:26:14,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:26:14,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:26:14,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36856.17 MB 2025-02-15 13:26:14,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40987.52 MB 2025-02-15 13:26:14,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:26:14,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41299.21 MB 2025-02-15 13:26:14,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49320.82 MB 2025-02-15 13:26:14,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 13:26:14,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46531.80 MB 2025-02-15 13:26:14,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:26:14,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:26:14,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:26:14,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41695.31 MB 2025-02-15 13:26:14,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42462.31 MB 2025-02-15 13:26:14,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:26:14,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49320.82 MB 2025-02-15 13:26:14,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49736.06 MB 2025-02-15 13:26:14,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:26:14,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43170.10 MB 2025-02-15 13:26:14,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:26:14,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:26:14,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:26:14,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42875.20 MB 2025-02-15 13:26:14,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43082.57 MB 2025-02-15 13:26:14,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.37 MB 2025-02-15 13:26:14,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49736.06 MB 2025-02-15 13:26:14,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49736.06 MB 2025-02-15 13:26:14,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:26:14,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43288.33 MB 2025-02-15 13:26:14,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:26:14,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:26:14,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.28 seconds 2025-02-15 13:26:14,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31638.69 MB 2025-02-15 13:26:14,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43283.64 MB 2025-02-15 13:26:14,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11644.95 MB 2025-02-15 13:26:14,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67700.26 MB 2025-02-15 13:26:14,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49736.06 MB 2025-02-15 13:26:14,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17964.20 MB 2025-02-15 13:26:14,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43288.33 MB 2025-02-15 13:26:14,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:26:14,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:26:14,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:26:14,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43283.64 MB 2025-02-15 13:26:14,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43384.11 MB 2025-02-15 13:26:14,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:26:14,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49736.06 MB 2025-02-15 13:26:14,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49736.06 MB 2025-02-15 13:26:14,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:26:14,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43986.91 MB 2025-02-15 13:26:14,480 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:26:14,480 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:26:14,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:26:14,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:26:14,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:26:14,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:26:14,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32901.61 MB 2025-02-15 13:26:14,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37096.10 MB 2025-02-15 13:26:14,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:26:14,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49736.06 MB 2025-02-15 13:26:14,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58126.76 MB 2025-02-15 13:26:14,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 13:26:14,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41290.40 MB 2025-02-15 13:26:14,645 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:26:14,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,647 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,648 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:26:14,652 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:26:14,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,653 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:26:14,653 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:26:14,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,654 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,655 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:14,660 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:26:14,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,661 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,661 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:14,662 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:26:14,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,662 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:14,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,662 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:26:14,663 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:26:14,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,663 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:14,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,666 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,667 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,668 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:26:14,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:14,676 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:51,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:51,898 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:26:51,903 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:26:51,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:51,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:26:51,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:26:51,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:27:17,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:27:17,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:27:17,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.99 seconds 2025-02-15 13:27:17,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:17,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39630.95 MB 2025-02-15 13:27:17,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45565.89 MB 2025-02-15 13:27:17,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-15 13:27:17,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66882.37 MB 2025-02-15 13:27:17,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58768.49 MB 2025-02-15 13:27:17,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8113.88 MB 2025-02-15 13:27:17,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54538.14 MB 2025-02-15 13:27:17,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:27:17,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:27:17,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:27:17,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:17,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45565.89 MB 2025-02-15 13:27:17,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39472.69 MB 2025-02-15 13:27:17,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-15 13:27:17,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58768.49 MB 2025-02-15 13:27:17,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70189.58 MB 2025-02-15 13:27:17,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11421.09 MB 2025-02-15 13:27:17,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62740.10 MB 2025-02-15 13:27:19,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:27:19,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:27:19,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 13:27:19,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:19,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39472.69 MB 2025-02-15 13:27:19,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40003.53 MB 2025-02-15 13:27:19,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:27:19,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70189.58 MB 2025-02-15 13:27:19,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52833.55 MB 2025-02-15 13:27:19,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17356.03 MB 2025-02-15 13:27:19,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43982.08 MB 2025-02-15 13:27:19,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:27:19,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:27:19,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:27:19,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:19,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40003.53 MB 2025-02-15 13:27:19,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41893.03 MB 2025-02-15 13:27:19,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:27:19,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 13:27:19,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52833.55 MB 2025-02-15 13:27:19,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:27:19,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43310.46 MB 2025-02-15 13:27:20,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:27:20,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:27:20,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:27:20,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41893.03 MB 2025-02-15 13:27:20,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44134.88 MB 2025-02-15 13:27:20,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:27:20,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 13:27:20,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52833.55 MB 2025-02-15 13:27:20,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:27:20,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49679.17 MB 2025-02-15 13:27:20,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:27:20,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:27:20,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:27:20,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40003.53 MB 2025-02-15 13:27:20,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44134.88 MB 2025-02-15 13:27:20,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:27:20,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 13:27:20,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52833.55 MB 2025-02-15 13:27:20,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:27:20,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49679.17 MB 2025-02-15 13:27:20,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:27:20,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:27:20,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 13:27:20,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44842.67 MB 2025-02-15 13:27:20,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40473.87 MB 2025-02-15 13:27:20,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4368.80 MB 2025-02-15 13:27:20,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52833.55 MB 2025-02-15 13:27:20,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53248.79 MB 2025-02-15 13:27:20,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:27:20,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46290.16 MB 2025-02-15 13:27:20,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:27:20,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:27:20,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:27:20,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40886.76 MB 2025-02-15 13:27:20,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41092.59 MB 2025-02-15 13:27:20,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.83 MB 2025-02-15 13:27:20,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53248.79 MB 2025-02-15 13:27:20,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53248.79 MB 2025-02-15 13:27:20,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:27:20,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41296.22 MB 2025-02-15 13:27:20,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:27:20,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:27:20,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.57 seconds 2025-02-15 13:27:20,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33788.15 MB 2025-02-15 13:27:20,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41292.85 MB 2025-02-15 13:27:20,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7504.70 MB 2025-02-15 13:27:20,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66882.37 MB 2025-02-15 13:27:20,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53248.79 MB 2025-02-15 13:27:20,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13633.59 MB 2025-02-15 13:27:20,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41296.22 MB 2025-02-15 13:27:20,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:27:20,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:27:20,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:27:20,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41292.85 MB 2025-02-15 13:27:20,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41392.91 MB 2025-02-15 13:27:20,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.06 MB 2025-02-15 13:27:20,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53248.79 MB 2025-02-15 13:27:20,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53248.79 MB 2025-02-15 13:27:20,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:27:20,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41993.28 MB 2025-02-15 13:27:20,762 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 13:27:20,762 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:27:20,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:27:20,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:27:20,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:27:20,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:27:20,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41392.91 MB 2025-02-15 13:27:20,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45570.47 MB 2025-02-15 13:27:20,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.56 MB 2025-02-15 13:27:20,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53248.79 MB 2025-02-15 13:27:20,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57426.31 MB 2025-02-15 13:27:20,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 13:27:20,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49747.99 MB 2025-02-15 13:27:20,931 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 13:27:20,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,933 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,934 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:27:20,939 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:27:20,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,940 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:27:20,940 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:27:20,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,940 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,941 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:27:20,947 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:27:20,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,947 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,948 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:27:20,948 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:27:20,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,948 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:27:20,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,949 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:27:20,949 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:27:20,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,949 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:27:20,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,953 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,954 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,954 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:27:20,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:27:20,963 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:13,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:13,908 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:13,913 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:28:13,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:13,915 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 842, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:28:13,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:13,916 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 842, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:28:27,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:28:27,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:28:27,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.10 seconds 2025-02-15 13:28:27,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:27,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28798.46 MB 2025-02-15 13:28:27,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31778.52 MB 2025-02-15 13:28:27,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2980.05 MB 2025-02-15 13:28:27,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66303.56 MB 2025-02-15 13:28:27,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38570.82 MB 2025-02-15 13:28:27,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27732.74 MB 2025-02-15 13:28:27,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40761.25 MB 2025-02-15 13:28:27,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:28:27,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:28:27,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 13:28:27,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:27,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31778.52 MB 2025-02-15 13:28:27,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30117.71 MB 2025-02-15 13:28:27,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1660.80 MB 2025-02-15 13:28:27,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38570.82 MB 2025-02-15 13:28:27,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45388.66 MB 2025-02-15 13:28:27,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6817.84 MB 2025-02-15 13:28:27,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41405.57 MB 2025-02-15 13:28:29,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:28:29,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:28:29,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:28:29,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30117.71 MB 2025-02-15 13:28:29,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30648.55 MB 2025-02-15 13:28:29,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:28:29,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45388.66 MB 2025-02-15 13:28:29,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37006.34 MB 2025-02-15 13:28:29,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8382.32 MB 2025-02-15 13:28:29,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34627.10 MB 2025-02-15 13:28:29,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:28:29,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:28:29,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:28:29,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30648.55 MB 2025-02-15 13:28:29,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32538.05 MB 2025-02-15 13:28:29,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:28:29,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37006.34 MB 2025-02-15 13:28:29,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37006.34 MB 2025-02-15 13:28:29,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:28:29,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33955.48 MB 2025-02-15 13:28:29,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:28:29,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:28:29,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:28:29,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32538.05 MB 2025-02-15 13:28:29,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34779.90 MB 2025-02-15 13:28:29,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:28:29,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37006.34 MB 2025-02-15 13:28:29,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42196.80 MB 2025-02-15 13:28:29,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:28:29,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40324.19 MB 2025-02-15 13:28:29,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:28:29,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:28:29,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:28:29,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30648.55 MB 2025-02-15 13:28:29,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34779.90 MB 2025-02-15 13:28:29,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:28:29,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37006.34 MB 2025-02-15 13:28:29,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42196.80 MB 2025-02-15 13:28:29,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:28:29,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40324.19 MB 2025-02-15 13:28:29,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:28:29,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:28:29,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:28:29,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35487.69 MB 2025-02-15 13:28:29,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36254.69 MB 2025-02-15 13:28:29,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:28:29,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42196.80 MB 2025-02-15 13:28:29,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-15 13:28:29,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:28:29,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36962.48 MB 2025-02-15 13:28:29,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:28:29,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:28:29,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:28:29,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36667.58 MB 2025-02-15 13:28:29,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36875.04 MB 2025-02-15 13:28:29,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.46 MB 2025-02-15 13:28:29,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42612.03 MB 2025-02-15 13:28:29,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-15 13:28:29,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:28:29,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37066.18 MB 2025-02-15 13:28:29,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:28:29,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:28:29,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.50 seconds 2025-02-15 13:28:29,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25864.87 MB 2025-02-15 13:28:29,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37076.11 MB 2025-02-15 13:28:29,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11211.24 MB 2025-02-15 13:28:29,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66303.56 MB 2025-02-15 13:28:29,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-15 13:28:29,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23691.53 MB 2025-02-15 13:28:29,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37076.11 MB 2025-02-15 13:28:29,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:28:29,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:28:29,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:28:29,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37076.11 MB 2025-02-15 13:28:29,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37176.58 MB 2025-02-15 13:28:29,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:28:29,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42612.03 MB 2025-02-15 13:28:29,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-15 13:28:29,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:28:29,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37779.38 MB 2025-02-15 13:28:29,702 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:28:29,702 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:28:29,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:28:29,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:28:29,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:28:29,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:28:29,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27127.79 MB 2025-02-15 13:28:29,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31322.27 MB 2025-02-15 13:28:29,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:28:29,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42612.03 MB 2025-02-15 13:28:29,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51002.74 MB 2025-02-15 13:28:29,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 13:28:29,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35516.58 MB 2025-02-15 13:28:29,868 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:28:29,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,870 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:28:29,875 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:28:29,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,876 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:28:29,876 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:28:29,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,877 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,878 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:29,883 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:28:29,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,884 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,885 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:29,885 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:28:29,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,885 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:29,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,886 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:28:29,886 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:28:29,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,886 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:28:29,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,889 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,890 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,891 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:28:29,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:28:29,899 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:26,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:26,172 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:26,177 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:29:26,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:26,178 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:29:26,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:26,179 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:29:45,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:29:45,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:29:45,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.54 seconds 2025-02-15 13:29:45,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:45,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31881.66 MB 2025-02-15 13:29:45,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36365.50 MB 2025-02-15 13:29:45,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-15 13:29:45,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60001.62 MB 2025-02-15 13:29:45,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48588.91 MB 2025-02-15 13:29:45,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11412.70 MB 2025-02-15 13:29:45,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45203.86 MB 2025-02-15 13:29:45,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:29:45,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:29:45,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:29:45,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:45,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36365.50 MB 2025-02-15 13:29:45,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32448.88 MB 2025-02-15 13:29:45,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-15 13:29:45,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48588.91 MB 2025-02-15 13:29:45,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57476.64 MB 2025-02-15 13:29:45,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8887.73 MB 2025-02-15 13:29:45,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49727.83 MB 2025-02-15 13:29:47,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:29:47,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:29:47,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:29:47,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:47,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32448.88 MB 2025-02-15 13:29:47,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32979.72 MB 2025-02-15 13:29:47,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:29:47,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57476.64 MB 2025-02-15 13:29:47,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39927.68 MB 2025-02-15 13:29:47,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17548.97 MB 2025-02-15 13:29:47,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36958.27 MB 2025-02-15 13:29:47,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:29:47,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:29:47,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:29:47,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:47,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32979.72 MB 2025-02-15 13:29:47,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34869.22 MB 2025-02-15 13:29:47,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:29:47,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39927.68 MB 2025-02-15 13:29:47,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39927.68 MB 2025-02-15 13:29:47,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:29:47,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36286.65 MB 2025-02-15 13:29:47,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:29:47,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:29:47,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:29:47,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:47,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34869.22 MB 2025-02-15 13:29:47,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37111.07 MB 2025-02-15 13:29:47,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:29:47,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39927.68 MB 2025-02-15 13:29:47,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-15 13:29:47,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:29:47,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42655.35 MB 2025-02-15 13:29:47,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:29:47,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:29:47,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:29:47,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:47,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32979.72 MB 2025-02-15 13:29:47,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37111.07 MB 2025-02-15 13:29:47,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:29:47,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39927.68 MB 2025-02-15 13:29:47,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-15 13:29:47,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:29:47,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42655.35 MB 2025-02-15 13:29:48,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:29:48,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:29:48,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:29:48,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:48,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37818.86 MB 2025-02-15 13:29:48,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38585.86 MB 2025-02-15 13:29:48,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:29:48,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44646.27 MB 2025-02-15 13:29:48,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-15 13:29:48,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:29:48,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39293.65 MB 2025-02-15 13:29:48,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:29:48,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:29:48,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:29:48,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:48,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38998.75 MB 2025-02-15 13:29:48,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39204.65 MB 2025-02-15 13:29:48,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.90 MB 2025-02-15 13:29:48,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-15 13:29:48,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-15 13:29:48,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:29:48,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39431.50 MB 2025-02-15 13:29:48,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:29:48,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:29:48,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.95 seconds 2025-02-15 13:29:48,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:48,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27467.33 MB 2025-02-15 13:29:48,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39405.21 MB 2025-02-15 13:29:48,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11937.88 MB 2025-02-15 13:29:48,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60001.62 MB 2025-02-15 13:29:48,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-15 13:29:48,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14938.01 MB 2025-02-15 13:29:48,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39431.50 MB 2025-02-15 13:29:48,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:29:48,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:29:48,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:29:48,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:48,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39405.21 MB 2025-02-15 13:29:48,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39505.42 MB 2025-02-15 13:29:48,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.21 MB 2025-02-15 13:29:48,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-15 13:29:48,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-15 13:29:48,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:29:48,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40107.11 MB 2025-02-15 13:29:48,417 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 13:29:48,417 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:29:48,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:29:48,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:29:48,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:29:48,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:29:48,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28729.73 MB 2025-02-15 13:29:48,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32913.44 MB 2025-02-15 13:29:48,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4183.71 MB 2025-02-15 13:29:48,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-15 13:29:48,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49247.42 MB 2025-02-15 13:29:48,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 13:29:48,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37097.26 MB 2025-02-15 13:29:48,582 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 13:29:48,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,585 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:29:48,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:29:48,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,590 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:29:48,590 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:29:48,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,591 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,592 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:48,598 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:29:48,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,598 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,599 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:48,599 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:29:48,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,599 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:48,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,600 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:29:48,600 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:29:48,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,600 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:29:48,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,603 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,604 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,605 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:29:48,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:29:48,613 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:37,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:37,742 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:37,750 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:30:37,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:37,753 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:30:37,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:37,755 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:30:56,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:30:56,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:30:56,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.07 seconds 2025-02-15 13:30:56,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:56,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31801.31 MB 2025-02-15 13:30:56,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36182.52 MB 2025-02-15 13:30:56,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-15 13:30:56,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58367.93 MB 2025-02-15 13:30:56,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49035.61 MB 2025-02-15 13:30:56,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9332.33 MB 2025-02-15 13:30:56,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45123.05 MB 2025-02-15 13:30:56,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:30:56,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:30:56,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:30:56,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:56,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36182.52 MB 2025-02-15 13:30:56,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32419.85 MB 2025-02-15 13:30:56,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-15 13:30:56,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49035.61 MB 2025-02-15 13:30:56,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55459.18 MB 2025-02-15 13:30:56,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6423.58 MB 2025-02-15 13:30:56,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49089.73 MB 2025-02-15 13:30:58,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:30:58,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:30:58,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:30:58,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:58,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32419.85 MB 2025-02-15 13:30:58,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32950.69 MB 2025-02-15 13:30:58,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:30:58,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55459.18 MB 2025-02-15 13:30:58,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49035.61 MB 2025-02-15 13:30:58,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6423.58 MB 2025-02-15 13:30:58,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36929.23 MB 2025-02-15 13:30:58,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:30:58,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:30:58,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:30:58,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:58,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32950.69 MB 2025-02-15 13:30:58,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34840.18 MB 2025-02-15 13:30:58,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:30:58,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49035.61 MB 2025-02-15 13:30:58,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49035.61 MB 2025-02-15 13:30:58,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:30:58,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36257.61 MB 2025-02-15 13:30:59,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:30:59,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:30:59,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:30:59,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34840.18 MB 2025-02-15 13:30:59,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37082.04 MB 2025-02-15 13:30:59,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:30:59,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49035.61 MB 2025-02-15 13:30:59,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49035.61 MB 2025-02-15 13:30:59,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:30:59,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42626.32 MB 2025-02-15 13:30:59,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:30:59,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:30:59,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:30:59,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32950.69 MB 2025-02-15 13:30:59,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37082.04 MB 2025-02-15 13:30:59,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:30:59,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49035.61 MB 2025-02-15 13:30:59,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49035.61 MB 2025-02-15 13:30:59,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:30:59,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42626.32 MB 2025-02-15 13:30:59,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:30:59,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:30:59,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:30:59,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37789.82 MB 2025-02-15 13:30:59,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38556.83 MB 2025-02-15 13:30:59,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:30:59,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49035.61 MB 2025-02-15 13:30:59,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49452.94 MB 2025-02-15 13:30:59,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:30:59,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39264.61 MB 2025-02-15 13:30:59,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:30:59,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:30:59,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:30:59,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38969.72 MB 2025-02-15 13:30:59,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39174.60 MB 2025-02-15 13:30:59,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.88 MB 2025-02-15 13:30:59,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49452.94 MB 2025-02-15 13:30:59,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49452.94 MB 2025-02-15 13:30:59,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:30:59,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39391.81 MB 2025-02-15 13:30:59,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:30:59,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:30:59,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.49 seconds 2025-02-15 13:30:59,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27488.02 MB 2025-02-15 13:30:59,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39374.64 MB 2025-02-15 13:30:59,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11886.62 MB 2025-02-15 13:30:59,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58367.93 MB 2025-02-15 13:30:59,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49452.94 MB 2025-02-15 13:30:59,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8914.99 MB 2025-02-15 13:30:59,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39391.81 MB 2025-02-15 13:30:59,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:30:59,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:30:59,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:30:59,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39374.64 MB 2025-02-15 13:30:59,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39474.59 MB 2025-02-15 13:30:59,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.95 MB 2025-02-15 13:30:59,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49452.94 MB 2025-02-15 13:30:59,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49452.94 MB 2025-02-15 13:30:59,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:30:59,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40075.33 MB 2025-02-15 13:30:59,529 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-15 13:30:59,530 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:30:59,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:30:59,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:30:59,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:30:59,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:30:59,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28749.90 MB 2025-02-15 13:30:59,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32922.84 MB 2025-02-15 13:30:59,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4172.94 MB 2025-02-15 13:30:59,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49452.94 MB 2025-02-15 13:30:59,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53626.27 MB 2025-02-15 13:30:59,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 13:30:59,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37096.18 MB 2025-02-15 13:30:59,698 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-15 13:30:59,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,699 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,700 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:30:59,705 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:30:59,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,706 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:30:59,706 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:30:59,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,707 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,707 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:59,713 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:30:59,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,714 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,714 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:59,714 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:30:59,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,715 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:59,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,715 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:30:59,715 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:30:59,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,716 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:30:59,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,720 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,720 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,721 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:30:59,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:30:59,730 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:31:43,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:31:43,125 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:31:43,130 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:31:43,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:31:43,131 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:31:43,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:31:43,132 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:32:02,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:32:02,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:32:02,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.13 seconds 2025-02-15 13:32:02,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:02,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31923.03 MB 2025-02-15 13:32:02,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36304.25 MB 2025-02-15 13:32:02,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-15 13:32:02,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62868.42 MB 2025-02-15 13:32:02,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49148.85 MB 2025-02-15 13:32:02,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13719.57 MB 2025-02-15 13:32:02,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45244.77 MB 2025-02-15 13:32:02,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:32:02,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:32:02,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:32:02,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:02,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36304.25 MB 2025-02-15 13:32:02,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32541.57 MB 2025-02-15 13:32:02,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-15 13:32:02,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49148.85 MB 2025-02-15 13:32:02,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55685.68 MB 2025-02-15 13:32:02,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6536.82 MB 2025-02-15 13:32:02,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49306.79 MB 2025-02-15 13:32:04,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:32:04,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:32:04,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:32:04,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32541.57 MB 2025-02-15 13:32:04,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33072.41 MB 2025-02-15 13:32:04,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:32:04,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55685.68 MB 2025-02-15 13:32:04,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44977.62 MB 2025-02-15 13:32:04,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10708.06 MB 2025-02-15 13:32:04,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37050.96 MB 2025-02-15 13:32:04,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:32:04,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:32:04,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:32:04,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33072.41 MB 2025-02-15 13:32:04,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34961.91 MB 2025-02-15 13:32:04,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:32:04,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44977.62 MB 2025-02-15 13:32:04,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44977.62 MB 2025-02-15 13:32:04,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:32:04,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36379.33 MB 2025-02-15 13:32:04,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:32:04,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:32:04,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:32:04,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34961.91 MB 2025-02-15 13:32:04,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37203.76 MB 2025-02-15 13:32:04,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:32:04,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44977.62 MB 2025-02-15 13:32:04,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44977.62 MB 2025-02-15 13:32:04,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:32:04,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42748.04 MB 2025-02-15 13:32:04,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:32:04,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:32:04,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:32:04,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33072.41 MB 2025-02-15 13:32:04,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37203.76 MB 2025-02-15 13:32:04,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:32:04,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44977.62 MB 2025-02-15 13:32:04,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44977.62 MB 2025-02-15 13:32:04,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:32:04,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42748.04 MB 2025-02-15 13:32:04,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:32:04,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:32:04,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:32:04,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37911.55 MB 2025-02-15 13:32:04,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38678.55 MB 2025-02-15 13:32:04,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:32:04,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44977.62 MB 2025-02-15 13:32:04,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45390.76 MB 2025-02-15 13:32:04,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 13:32:04,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39386.34 MB 2025-02-15 13:32:04,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:32:04,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:32:04,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:32:04,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39091.44 MB 2025-02-15 13:32:04,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39297.13 MB 2025-02-15 13:32:04,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.69 MB 2025-02-15 13:32:04,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45390.76 MB 2025-02-15 13:32:04,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45390.76 MB 2025-02-15 13:32:04,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:32:04,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39513.44 MB 2025-02-15 13:32:04,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:32:04,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:32:04,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.53 seconds 2025-02-15 13:32:04,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27609.74 MB 2025-02-15 13:32:04,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39498.21 MB 2025-02-15 13:32:04,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11888.46 MB 2025-02-15 13:32:04,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62868.42 MB 2025-02-15 13:32:04,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45390.76 MB 2025-02-15 13:32:04,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17477.66 MB 2025-02-15 13:32:04,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39513.44 MB 2025-02-15 13:32:04,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:32:04,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:32:04,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:32:04,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39498.21 MB 2025-02-15 13:32:04,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39598.67 MB 2025-02-15 13:32:04,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:32:04,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45390.76 MB 2025-02-15 13:32:04,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45390.76 MB 2025-02-15 13:32:04,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:32:04,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40201.47 MB 2025-02-15 13:32:04,953 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:32:04,953 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:32:04,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:32:04,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:32:04,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:32:04,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:32:04,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28872.66 MB 2025-02-15 13:32:04,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33067.15 MB 2025-02-15 13:32:04,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:32:04,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45390.76 MB 2025-02-15 13:32:04,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49585.06 MB 2025-02-15 13:32:04,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:32:04,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37261.45 MB 2025-02-15 13:32:05,117 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:32:05,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,119 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,120 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:32:05,124 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:32:05,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,125 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:32:05,125 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:32:05,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,126 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,127 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:05,132 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:32:05,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,133 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,133 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:05,134 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:32:05,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,134 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:05,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,134 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:32:05,135 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:32:05,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,135 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:05,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,139 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,139 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,140 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:32:05,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:05,149 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:56,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:56,363 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:32:56,371 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:32:56,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:56,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 848, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:32:56,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:32:56,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 848, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:33:09,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:33:09,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:33:09,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.16 seconds 2025-02-15 13:33:09,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:09,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29327.18 MB 2025-02-15 13:33:09,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32328.20 MB 2025-02-15 13:33:09,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3001.02 MB 2025-02-15 13:33:09,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58948.85 MB 2025-02-15 13:33:09,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39099.30 MB 2025-02-15 13:33:09,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19849.54 MB 2025-02-15 13:33:09,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41289.96 MB 2025-02-15 13:33:09,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:33:09,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:33:09,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 13:33:09,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:09,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32328.20 MB 2025-02-15 13:33:09,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30636.86 MB 2025-02-15 13:33:09,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1691.34 MB 2025-02-15 13:33:09,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39099.30 MB 2025-02-15 13:33:09,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46756.00 MB 2025-02-15 13:33:09,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7656.70 MB 2025-02-15 13:33:09,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42081.16 MB 2025-02-15 13:33:11,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:33:11,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:33:11,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:33:11,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30636.86 MB 2025-02-15 13:33:11,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31167.70 MB 2025-02-15 13:33:11,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:33:11,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46756.00 MB 2025-02-15 13:33:11,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38222.69 MB 2025-02-15 13:33:11,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8533.31 MB 2025-02-15 13:33:11,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35146.25 MB 2025-02-15 13:33:11,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:33:11,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:33:11,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:33:11,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31167.70 MB 2025-02-15 13:33:11,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33057.19 MB 2025-02-15 13:33:11,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:33:11,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38222.69 MB 2025-02-15 13:33:11,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38222.69 MB 2025-02-15 13:33:11,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:33:11,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34474.62 MB 2025-02-15 13:33:11,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:33:11,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:33:11,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:33:11,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33057.19 MB 2025-02-15 13:33:11,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35299.05 MB 2025-02-15 13:33:11,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:33:11,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38222.69 MB 2025-02-15 13:33:11,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42941.28 MB 2025-02-15 13:33:11,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:33:11,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40843.33 MB 2025-02-15 13:33:11,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:33:11,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:33:11,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:33:11,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31167.70 MB 2025-02-15 13:33:11,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35299.05 MB 2025-02-15 13:33:11,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:33:11,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38222.69 MB 2025-02-15 13:33:11,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42941.28 MB 2025-02-15 13:33:11,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:33:11,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40843.33 MB 2025-02-15 13:33:11,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:33:11,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:33:11,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 13:33:11,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36006.84 MB 2025-02-15 13:33:11,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36773.84 MB 2025-02-15 13:33:11,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:33:11,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42941.28 MB 2025-02-15 13:33:11,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43358.62 MB 2025-02-15 13:33:11,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:33:11,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37481.63 MB 2025-02-15 13:33:11,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:33:11,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:33:11,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:33:11,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37186.73 MB 2025-02-15 13:33:11,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37391.61 MB 2025-02-15 13:33:11,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.88 MB 2025-02-15 13:33:11,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43358.62 MB 2025-02-15 13:33:11,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43358.62 MB 2025-02-15 13:33:11,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:33:11,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37573.88 MB 2025-02-15 13:33:11,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:33:11,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:33:11,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.61 seconds 2025-02-15 13:33:11,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:11,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26372.68 MB 2025-02-15 13:33:11,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37592.19 MB 2025-02-15 13:33:11,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11219.51 MB 2025-02-15 13:33:11,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58948.85 MB 2025-02-15 13:33:11,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43358.62 MB 2025-02-15 13:33:11,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15590.23 MB 2025-02-15 13:33:11,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37592.19 MB 2025-02-15 13:33:12,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:33:12,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:33:12,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:33:12,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:12,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37592.19 MB 2025-02-15 13:33:12,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37692.41 MB 2025-02-15 13:33:12,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-15 13:33:12,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43358.62 MB 2025-02-15 13:33:12,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43358.62 MB 2025-02-15 13:33:12,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:33:12,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38293.73 MB 2025-02-15 13:33:12,287 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 13:33:12,288 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:33:12,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:33:12,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:33:12,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:33:12,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:33:12,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27635.11 MB 2025-02-15 13:33:12,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31819.33 MB 2025-02-15 13:33:12,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-15 13:33:12,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43358.62 MB 2025-02-15 13:33:12,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51728.35 MB 2025-02-15 13:33:12,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-15 13:33:12,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36003.15 MB 2025-02-15 13:33:12,560 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 13:33:12,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,564 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:33:12,572 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:33:12,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,574 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:33:12,574 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:33:12,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,576 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,577 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:33:12,587 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:33:12,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,588 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,589 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:33:12,589 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:33:12,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,590 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:33:12,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,591 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:33:12,591 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:33:12,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,592 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:33:12,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,603 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,606 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,609 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:33:12,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:33:12,620 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:09,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:09,921 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:09,927 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:34:09,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:09,928 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 982, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:34:09,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:09,929 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 982, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:34:25,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:34:25,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:34:25,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.19 seconds 2025-02-15 13:34:25,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:25,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30383.68 MB 2025-02-15 13:34:25,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33858.92 MB 2025-02-15 13:34:25,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3475.24 MB 2025-02-15 13:34:25,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61213.77 MB 2025-02-15 13:34:25,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39694.89 MB 2025-02-15 13:34:25,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21518.88 MB 2025-02-15 13:34:25,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42799.45 MB 2025-02-15 13:34:25,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:34:25,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:34:25,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:34:25,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:25,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33858.92 MB 2025-02-15 13:34:25,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31455.20 MB 2025-02-15 13:34:25,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2403.72 MB 2025-02-15 13:34:25,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39694.89 MB 2025-02-15 13:34:25,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47699.72 MB 2025-02-15 13:34:25,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8004.83 MB 2025-02-15 13:34:25,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44671.78 MB 2025-02-15 13:34:27,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:34:27,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:34:27,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 13:34:27,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31455.20 MB 2025-02-15 13:34:27,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31986.04 MB 2025-02-15 13:34:27,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:34:27,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47699.72 MB 2025-02-15 13:34:27,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37633.39 MB 2025-02-15 13:34:27,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10066.33 MB 2025-02-15 13:34:27,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35964.59 MB 2025-02-15 13:34:27,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:34:27,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:34:27,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:34:27,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31986.04 MB 2025-02-15 13:34:27,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33875.14 MB 2025-02-15 13:34:27,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.09 MB 2025-02-15 13:34:27,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37633.39 MB 2025-02-15 13:34:27,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37633.39 MB 2025-02-15 13:34:27,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:34:27,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35292.57 MB 2025-02-15 13:34:27,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:34:27,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:34:27,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:34:27,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33875.14 MB 2025-02-15 13:34:27,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36116.99 MB 2025-02-15 13:34:27,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:34:27,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37633.39 MB 2025-02-15 13:34:27,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43767.56 MB 2025-02-15 13:34:27,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:34:27,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41661.27 MB 2025-02-15 13:34:27,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:34:27,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:34:27,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:34:27,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31986.04 MB 2025-02-15 13:34:27,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36116.99 MB 2025-02-15 13:34:27,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.95 MB 2025-02-15 13:34:27,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37633.39 MB 2025-02-15 13:34:27,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43767.56 MB 2025-02-15 13:34:27,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 13:34:27,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41661.27 MB 2025-02-15 13:34:27,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:34:27,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:34:27,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:34:27,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36824.78 MB 2025-02-15 13:34:27,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37591.78 MB 2025-02-15 13:34:27,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:34:27,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43767.56 MB 2025-02-15 13:34:27,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44184.90 MB 2025-02-15 13:34:27,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:34:27,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38299.57 MB 2025-02-15 13:34:27,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:34:27,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:34:27,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:34:27,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38004.67 MB 2025-02-15 13:34:27,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38211.49 MB 2025-02-15 13:34:27,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.81 MB 2025-02-15 13:34:27,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44184.90 MB 2025-02-15 13:34:27,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44184.90 MB 2025-02-15 13:34:27,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:34:27,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38418.29 MB 2025-02-15 13:34:27,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:34:27,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:34:27,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.58 seconds 2025-02-15 13:34:27,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26962.31 MB 2025-02-15 13:34:27,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38412.07 MB 2025-02-15 13:34:27,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11449.76 MB 2025-02-15 13:34:27,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61213.77 MB 2025-02-15 13:34:27,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44184.90 MB 2025-02-15 13:34:27,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17028.87 MB 2025-02-15 13:34:27,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38418.29 MB 2025-02-15 13:34:27,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:34:27,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:34:27,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:34:27,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38412.07 MB 2025-02-15 13:34:27,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38512.29 MB 2025-02-15 13:34:27,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-15 13:34:27,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44184.90 MB 2025-02-15 13:34:27,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44184.90 MB 2025-02-15 13:34:27,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:34:27,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39113.61 MB 2025-02-15 13:34:27,794 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 13:34:27,794 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:34:27,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:34:27,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:34:27,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:34:27,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:34:27,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28224.74 MB 2025-02-15 13:34:27,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32408.97 MB 2025-02-15 13:34:27,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-15 13:34:27,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44184.90 MB 2025-02-15 13:34:27,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52554.63 MB 2025-02-15 13:34:27,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-15 13:34:27,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36592.78 MB 2025-02-15 13:34:27,961 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 13:34:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:34:27,968 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:34:27,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,969 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:34:27,969 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:34:27,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,970 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:27,976 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:34:27,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,976 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,977 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:27,977 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:34:27,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,977 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:27,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,978 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:34:27,978 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:34:27,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,978 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:34:27,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,982 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,983 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,983 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:34:27,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:34:27,992 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:25,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:25,757 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:25,762 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:35:25,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:25,764 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:35:25,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:25,765 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:35:45,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:35:45,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:35:45,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.82 seconds 2025-02-15 13:35:45,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:45,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32601.78 MB 2025-02-15 13:35:45,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37142.24 MB 2025-02-15 13:35:45,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4540.47 MB 2025-02-15 13:35:45,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62161.68 MB 2025-02-15 13:35:45,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49251.61 MB 2025-02-15 13:35:45,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12910.07 MB 2025-02-15 13:35:45,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46150.01 MB 2025-02-15 13:35:45,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:35:45,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:35:45,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:35:45,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:45,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37142.24 MB 2025-02-15 13:35:45,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33140.69 MB 2025-02-15 13:35:45,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4001.55 MB 2025-02-15 13:35:45,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49251.61 MB 2025-02-15 13:35:45,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58265.17 MB 2025-02-15 13:35:45,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9013.56 MB 2025-02-15 13:35:45,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50694.34 MB 2025-02-15 13:35:47,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:35:47,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:35:47,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:35:47,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33140.69 MB 2025-02-15 13:35:47,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33671.53 MB 2025-02-15 13:35:47,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:35:47,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58265.17 MB 2025-02-15 13:35:47,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 13:35:47,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17729.32 MB 2025-02-15 13:35:47,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37650.08 MB 2025-02-15 13:35:47,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:35:47,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:35:47,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:35:47,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33671.53 MB 2025-02-15 13:35:47,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35560.67 MB 2025-02-15 13:35:47,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.14 MB 2025-02-15 13:35:47,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 13:35:47,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40535.85 MB 2025-02-15 13:35:47,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:35:47,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36978.10 MB 2025-02-15 13:35:47,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:35:47,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:35:47,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:35:47,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35560.67 MB 2025-02-15 13:35:47,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37802.53 MB 2025-02-15 13:35:47,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:35:47,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 13:35:47,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45726.30 MB 2025-02-15 13:35:47,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:35:47,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43346.81 MB 2025-02-15 13:35:47,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:35:47,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:35:47,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:35:47,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33671.53 MB 2025-02-15 13:35:47,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37802.53 MB 2025-02-15 13:35:47,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.00 MB 2025-02-15 13:35:47,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40535.85 MB 2025-02-15 13:35:47,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45726.30 MB 2025-02-15 13:35:47,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 13:35:47,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43346.81 MB 2025-02-15 13:35:47,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:35:47,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:35:47,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:35:47,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38510.32 MB 2025-02-15 13:35:47,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39277.32 MB 2025-02-15 13:35:47,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:35:47,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45726.30 MB 2025-02-15 13:35:47,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46145.73 MB 2025-02-15 13:35:47,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 13:35:47,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39985.11 MB 2025-02-15 13:35:47,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:35:47,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:35:47,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:35:47,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39690.21 MB 2025-02-15 13:35:47,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39896.78 MB 2025-02-15 13:35:47,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.57 MB 2025-02-15 13:35:47,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46145.73 MB 2025-02-15 13:35:47,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46145.73 MB 2025-02-15 13:35:47,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:35:47,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40114.60 MB 2025-02-15 13:35:47,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:35:47,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:35:47,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.23 seconds 2025-02-15 13:35:47,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:47,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28131.70 MB 2025-02-15 13:35:47,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40097.70 MB 2025-02-15 13:35:47,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11966.00 MB 2025-02-15 13:35:47,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62161.68 MB 2025-02-15 13:35:47,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46145.73 MB 2025-02-15 13:35:47,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16015.95 MB 2025-02-15 13:35:47,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40114.60 MB 2025-02-15 13:35:48,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:35:48,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:35:48,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:35:48,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:48,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40097.70 MB 2025-02-15 13:35:48,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40198.09 MB 2025-02-15 13:35:48,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.39 MB 2025-02-15 13:35:48,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46145.73 MB 2025-02-15 13:35:48,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46145.73 MB 2025-02-15 13:35:48,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:35:48,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40800.45 MB 2025-02-15 13:35:48,282 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 13:35:48,282 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:35:48,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:35:48,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:35:48,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:35:48,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:35:48,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29394.48 MB 2025-02-15 13:35:48,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33586.68 MB 2025-02-15 13:35:48,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.21 MB 2025-02-15 13:35:48,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46145.73 MB 2025-02-15 13:35:48,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54530.15 MB 2025-02-15 13:35:48,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 13:35:48,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37777.58 MB 2025-02-15 13:35:48,446 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 13:35:48,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,447 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,448 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:35:48,453 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:35:48,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,454 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:35:48,454 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:35:48,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,455 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,455 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:48,461 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:35:48,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,462 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,462 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:48,462 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:35:48,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,463 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:48,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,463 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:35:48,463 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:35:48,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,464 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:35:48,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,467 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,468 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,469 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:35:48,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:35:48,478 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:36:52,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:36:52,144 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:36:52,149 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:36:52,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:36:52,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1256, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:36:52,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:36:52,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1256, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:37:11,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:37:11,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:37:11,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.35 seconds 2025-02-15 13:37:11,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:11,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32536.41 MB 2025-02-15 13:37:11,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36982.37 MB 2025-02-15 13:37:11,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4445.96 MB 2025-02-15 13:37:11,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64258.83 MB 2025-02-15 13:37:11,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49293.56 MB 2025-02-15 13:37:11,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14965.28 MB 2025-02-15 13:37:11,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45858.15 MB 2025-02-15 13:37:11,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:37:11,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:37:11,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:37:11,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:11,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36982.37 MB 2025-02-15 13:37:11,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33123.10 MB 2025-02-15 13:37:11,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3859.27 MB 2025-02-15 13:37:11,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49293.56 MB 2025-02-15 13:37:11,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58068.04 MB 2025-02-15 13:37:11,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8774.48 MB 2025-02-15 13:37:11,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50175.29 MB 2025-02-15 13:37:13,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:37:13,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:37:13,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:37:13,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33123.10 MB 2025-02-15 13:37:13,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33653.94 MB 2025-02-15 13:37:13,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:37:13,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58068.04 MB 2025-02-15 13:37:13,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44847.60 MB 2025-02-15 13:37:13,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13220.45 MB 2025-02-15 13:37:13,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37632.49 MB 2025-02-15 13:37:13,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:37:13,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:37:13,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:37:13,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33653.94 MB 2025-02-15 13:37:13,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35543.13 MB 2025-02-15 13:37:13,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.19 MB 2025-02-15 13:37:13,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44847.60 MB 2025-02-15 13:37:13,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44847.60 MB 2025-02-15 13:37:13,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:13,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36960.56 MB 2025-02-15 13:37:13,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:37:13,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:37:13,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:37:13,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35543.13 MB 2025-02-15 13:37:13,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37784.99 MB 2025-02-15 13:37:13,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:37:13,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44847.60 MB 2025-02-15 13:37:13,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44847.60 MB 2025-02-15 13:37:13,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:13,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43329.27 MB 2025-02-15 13:37:13,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:37:13,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:37:13,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:37:13,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33653.94 MB 2025-02-15 13:37:13,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37784.99 MB 2025-02-15 13:37:13,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.05 MB 2025-02-15 13:37:13,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44847.60 MB 2025-02-15 13:37:13,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44847.60 MB 2025-02-15 13:37:13,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:13,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43329.27 MB 2025-02-15 13:37:13,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:37:13,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:37:13,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:37:13,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38492.78 MB 2025-02-15 13:37:13,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39259.78 MB 2025-02-15 13:37:13,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:37:13,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44847.60 MB 2025-02-15 13:37:13,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45264.93 MB 2025-02-15 13:37:13,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:37:13,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39967.57 MB 2025-02-15 13:37:13,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:37:13,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:37:13,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:37:13,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39672.67 MB 2025-02-15 13:37:13,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39878.03 MB 2025-02-15 13:37:13,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.36 MB 2025-02-15 13:37:13,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45264.93 MB 2025-02-15 13:37:13,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45264.93 MB 2025-02-15 13:37:13,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:13,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40101.88 MB 2025-02-15 13:37:13,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:37:13,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:37:13,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.75 seconds 2025-02-15 13:37:13,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:13,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28160.41 MB 2025-02-15 13:37:13,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40077.90 MB 2025-02-15 13:37:13,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11917.49 MB 2025-02-15 13:37:13,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64258.83 MB 2025-02-15 13:37:13,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45264.93 MB 2025-02-15 13:37:13,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18993.91 MB 2025-02-15 13:37:13,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40101.88 MB 2025-02-15 13:37:14,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:37:14,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:37:14,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:37:14,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:14,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40077.90 MB 2025-02-15 13:37:14,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40177.76 MB 2025-02-15 13:37:14,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.86 MB 2025-02-15 13:37:14,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45264.93 MB 2025-02-15 13:37:14,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45264.93 MB 2025-02-15 13:37:14,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:14,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40776.95 MB 2025-02-15 13:37:14,187 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 13:37:14,188 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:37:14,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:37:14,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:37:14,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:37:14,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:37:14,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29422.12 MB 2025-02-15 13:37:14,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33591.47 MB 2025-02-15 13:37:14,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4169.35 MB 2025-02-15 13:37:14,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45264.93 MB 2025-02-15 13:37:14,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45264.93 MB 2025-02-15 13:37:14,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:37:14,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37760.30 MB 2025-02-15 13:37:14,351 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 13:37:14,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,352 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,353 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:37:14,358 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:37:14,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,359 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:37:14,359 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:37:14,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,360 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,360 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:37:14,366 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:37:14,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,367 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,367 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:37:14,367 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:37:14,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,368 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:37:14,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,368 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:37:14,368 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:37:14,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,369 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:37:14,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,372 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,373 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,374 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:37:14,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:37:14,383 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:01,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:01,772 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:01,777 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:38:01,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:01,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1330, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:38:01,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:01,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1330, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:38:22,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:38:22,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:38:22,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.56 seconds 2025-02-15 13:38:22,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:22,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33173.72 MB 2025-02-15 13:38:22,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37880.51 MB 2025-02-15 13:38:22,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4706.80 MB 2025-02-15 13:38:22,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55115.25 MB 2025-02-15 13:38:22,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50505.71 MB 2025-02-15 13:38:22,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4609.54 MB 2025-02-15 13:38:22,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46721.95 MB 2025-02-15 13:38:22,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:38:22,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:38:22,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:38:22,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:22,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37880.51 MB 2025-02-15 13:38:22,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33629.46 MB 2025-02-15 13:38:22,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4251.05 MB 2025-02-15 13:38:22,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50505.71 MB 2025-02-15 13:38:22,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57443.09 MB 2025-02-15 13:38:22,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6937.38 MB 2025-02-15 13:38:22,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51824.07 MB 2025-02-15 13:38:24,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:38:24,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:38:24,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:38:24,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33629.46 MB 2025-02-15 13:38:24,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34160.31 MB 2025-02-15 13:38:24,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:38:24,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57443.09 MB 2025-02-15 13:38:24,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46319.80 MB 2025-02-15 13:38:24,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11123.29 MB 2025-02-15 13:38:24,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38138.85 MB 2025-02-15 13:38:24,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:38:24,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:38:24,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:38:24,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34160.31 MB 2025-02-15 13:38:24,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36049.54 MB 2025-02-15 13:38:24,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.24 MB 2025-02-15 13:38:24,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46319.80 MB 2025-02-15 13:38:24,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46319.80 MB 2025-02-15 13:38:24,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:38:24,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37466.97 MB 2025-02-15 13:38:24,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:38:24,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:38:24,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:38:24,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36049.54 MB 2025-02-15 13:38:24,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38291.40 MB 2025-02-15 13:38:24,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:38:24,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46319.80 MB 2025-02-15 13:38:24,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46319.80 MB 2025-02-15 13:38:24,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:38:24,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43835.68 MB 2025-02-15 13:38:24,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:38:24,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:38:24,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:38:24,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34160.31 MB 2025-02-15 13:38:24,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38291.40 MB 2025-02-15 13:38:24,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.09 MB 2025-02-15 13:38:24,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46319.80 MB 2025-02-15 13:38:24,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46319.80 MB 2025-02-15 13:38:24,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:38:24,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43835.68 MB 2025-02-15 13:38:24,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:38:24,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:38:24,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:38:24,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38999.19 MB 2025-02-15 13:38:24,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39766.19 MB 2025-02-15 13:38:24,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:38:24,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46319.80 MB 2025-02-15 13:38:24,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46737.13 MB 2025-02-15 13:38:24,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:38:24,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40473.98 MB 2025-02-15 13:38:24,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:38:24,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:38:24,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:38:24,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40179.08 MB 2025-02-15 13:38:24,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40385.32 MB 2025-02-15 13:38:24,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.24 MB 2025-02-15 13:38:24,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46737.13 MB 2025-02-15 13:38:24,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46737.13 MB 2025-02-15 13:38:24,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:38:24,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40609.87 MB 2025-02-15 13:38:24,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:38:24,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:38:24,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.98 seconds 2025-02-15 13:38:24,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:24,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28539.89 MB 2025-02-15 13:38:24,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40586.34 MB 2025-02-15 13:38:24,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12046.45 MB 2025-02-15 13:38:24,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55115.25 MB 2025-02-15 13:38:24,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46737.13 MB 2025-02-15 13:38:24,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8378.12 MB 2025-02-15 13:38:24,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40609.87 MB 2025-02-15 13:38:25,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:38:25,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:38:25,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:38:25,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:25,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40586.34 MB 2025-02-15 13:38:25,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40686.78 MB 2025-02-15 13:38:25,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-15 13:38:25,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46737.13 MB 2025-02-15 13:38:25,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46737.13 MB 2025-02-15 13:38:25,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:38:25,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41289.43 MB 2025-02-15 13:38:25,047 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 13:38:25,047 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for the video is 2.'] 2025-02-15 13:38:25,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:38:25,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:38:25,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:38:25,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:38:25,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29802.76 MB 2025-02-15 13:38:25,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33996.22 MB 2025-02-15 13:38:25,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4193.46 MB 2025-02-15 13:38:25,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46737.13 MB 2025-02-15 13:38:25,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50931.43 MB 2025-02-15 13:38:25,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:38:25,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38189.17 MB 2025-02-15 13:38:25,211 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 13:38:25,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,212 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,213 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:38:25,218 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:38:25,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,219 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:38:25,219 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for the video is 2.'] 2025-02-15 13:38:25,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,220 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,220 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:25,226 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:38:25,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,227 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,227 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:25,227 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:38:25,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,228 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:25,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,228 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:38:25,228 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:38:25,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,229 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:38:25,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,233 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,235 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,236 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:38:25,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:38:25,246 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:29,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:29,910 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:29,915 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:39:29,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:29,916 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1268, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:39:29,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:29,917 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1268, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:39:49,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:39:49,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:39:49,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.51 seconds 2025-02-15 13:39:49,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:49,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32862.43 MB 2025-02-15 13:39:49,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37349.81 MB 2025-02-15 13:39:49,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4487.38 MB 2025-02-15 13:39:49,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60903.39 MB 2025-02-15 13:39:49,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50635.74 MB 2025-02-15 13:39:49,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10267.66 MB 2025-02-15 13:39:49,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46184.18 MB 2025-02-15 13:39:49,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:39:49,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:39:49,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:39:49,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:49,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37349.81 MB 2025-02-15 13:39:49,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33427.89 MB 2025-02-15 13:39:49,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3921.93 MB 2025-02-15 13:39:49,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50635.74 MB 2025-02-15 13:39:49,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59538.15 MB 2025-02-15 13:39:49,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8902.41 MB 2025-02-15 13:39:49,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50732.99 MB 2025-02-15 13:39:51,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:39:51,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:39:51,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:39:51,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33427.89 MB 2025-02-15 13:39:51,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33958.73 MB 2025-02-15 13:39:51,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:39:51,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59538.15 MB 2025-02-15 13:39:51,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46451.92 MB 2025-02-15 13:39:51,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13086.23 MB 2025-02-15 13:39:51,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37937.28 MB 2025-02-15 13:39:51,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:39:51,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:39:51,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:39:51,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33958.73 MB 2025-02-15 13:39:51,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35848.01 MB 2025-02-15 13:39:51,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.29 MB 2025-02-15 13:39:51,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46451.92 MB 2025-02-15 13:39:51,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46451.92 MB 2025-02-15 13:39:51,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:51,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37265.44 MB 2025-02-15 13:39:51,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:39:51,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:39:51,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:39:51,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35848.01 MB 2025-02-15 13:39:51,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38089.87 MB 2025-02-15 13:39:51,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:39:51,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46451.92 MB 2025-02-15 13:39:51,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46451.92 MB 2025-02-15 13:39:51,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:51,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43634.15 MB 2025-02-15 13:39:51,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:39:51,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:39:51,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:39:51,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33958.73 MB 2025-02-15 13:39:51,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38089.87 MB 2025-02-15 13:39:51,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.14 MB 2025-02-15 13:39:51,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46451.92 MB 2025-02-15 13:39:51,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46451.92 MB 2025-02-15 13:39:51,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:51,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43634.15 MB 2025-02-15 13:39:51,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:39:51,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:39:51,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:39:51,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38797.66 MB 2025-02-15 13:39:51,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39564.66 MB 2025-02-15 13:39:51,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:39:51,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46451.92 MB 2025-02-15 13:39:51,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46869.25 MB 2025-02-15 13:39:51,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:39:51,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40272.45 MB 2025-02-15 13:39:51,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:39:51,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:39:51,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:39:51,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39977.55 MB 2025-02-15 13:39:51,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40183.66 MB 2025-02-15 13:39:51,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.11 MB 2025-02-15 13:39:51,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46869.25 MB 2025-02-15 13:39:51,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46869.25 MB 2025-02-15 13:39:51,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:51,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40412.04 MB 2025-02-15 13:39:51,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:39:51,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:39:51,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.92 seconds 2025-02-15 13:39:51,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:51,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28444.62 MB 2025-02-15 13:39:51,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40384.29 MB 2025-02-15 13:39:51,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11939.67 MB 2025-02-15 13:39:51,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60903.39 MB 2025-02-15 13:39:51,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46869.25 MB 2025-02-15 13:39:51,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14034.14 MB 2025-02-15 13:39:51,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40412.04 MB 2025-02-15 13:39:52,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:39:52,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:39:52,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:39:52,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:52,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40384.29 MB 2025-02-15 13:39:52,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40484.54 MB 2025-02-15 13:39:52,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.25 MB 2025-02-15 13:39:52,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46869.25 MB 2025-02-15 13:39:52,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46869.25 MB 2025-02-15 13:39:52,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:52,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41086.01 MB 2025-02-15 13:39:52,129 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 13:39:52,129 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:39:52,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:39:52,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:39:52,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:39:52,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:39:52,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29707.10 MB 2025-02-15 13:39:52,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33892.35 MB 2025-02-15 13:39:52,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.25 MB 2025-02-15 13:39:52,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46869.25 MB 2025-02-15 13:39:52,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46869.25 MB 2025-02-15 13:39:52,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:39:52,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38077.09 MB 2025-02-15 13:39:52,299 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 13:39:52,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,300 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,301 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:39:52,306 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:39:52,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,307 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:39:52,307 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:39:52,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,308 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,308 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:52,314 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:39:52,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,315 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,315 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:52,315 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:39:52,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,316 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:52,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,316 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:39:52,316 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:39:52,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,317 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:39:52,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,321 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,323 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,324 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:39:52,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:39:52,334 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:40:42,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:40:42,449 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:40:42,454 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:40:42,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:40:42,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1334, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:40:42,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:40:42,457 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1334, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:41:03,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:41:03,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:41:03,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.60 seconds 2025-02-15 13:41:03,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:03,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33444.91 MB 2025-02-15 13:41:03,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38165.86 MB 2025-02-15 13:41:03,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4720.95 MB 2025-02-15 13:41:03,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56962.84 MB 2025-02-15 13:41:03,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47104.13 MB 2025-02-15 13:41:03,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9858.71 MB 2025-02-15 13:41:03,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46993.14 MB 2025-02-15 13:41:03,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:41:03,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:41:03,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 13:41:03,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:03,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38165.86 MB 2025-02-15 13:41:03,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33893.58 MB 2025-02-15 13:41:03,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4272.28 MB 2025-02-15 13:41:03,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47104.13 MB 2025-02-15 13:41:03,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56476.30 MB 2025-02-15 13:41:03,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9372.17 MB 2025-02-15 13:41:03,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52257.19 MB 2025-02-15 13:41:05,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:41:05,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:41:05,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:41:05,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33893.58 MB 2025-02-15 13:41:05,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34424.42 MB 2025-02-15 13:41:05,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:41:05,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56476.30 MB 2025-02-15 13:41:05,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42381.34 MB 2025-02-15 13:41:05,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14094.96 MB 2025-02-15 13:41:05,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38402.97 MB 2025-02-15 13:41:05,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:41:05,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:41:05,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:41:05,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34424.42 MB 2025-02-15 13:41:05,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36313.75 MB 2025-02-15 13:41:05,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.33 MB 2025-02-15 13:41:05,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42381.34 MB 2025-02-15 13:41:05,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42381.34 MB 2025-02-15 13:41:05,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:41:05,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37731.18 MB 2025-02-15 13:41:05,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:41:05,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:41:05,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:41:05,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36313.75 MB 2025-02-15 13:41:05,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38555.61 MB 2025-02-15 13:41:05,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:41:05,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42381.34 MB 2025-02-15 13:41:05,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46628.08 MB 2025-02-15 13:41:05,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 13:41:05,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44099.89 MB 2025-02-15 13:41:05,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:41:05,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:41:05,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:41:05,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34424.42 MB 2025-02-15 13:41:05,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38555.61 MB 2025-02-15 13:41:05,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.19 MB 2025-02-15 13:41:05,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42381.34 MB 2025-02-15 13:41:05,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46628.08 MB 2025-02-15 13:41:05,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 13:41:05,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44099.89 MB 2025-02-15 13:41:05,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:41:05,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:41:05,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:41:05,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39263.40 MB 2025-02-15 13:41:05,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40030.40 MB 2025-02-15 13:41:05,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:41:05,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46628.08 MB 2025-02-15 13:41:05,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47045.41 MB 2025-02-15 13:41:05,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:41:05,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40738.19 MB 2025-02-15 13:41:05,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:41:05,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:41:05,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:41:05,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40443.29 MB 2025-02-15 13:41:05,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40649.60 MB 2025-02-15 13:41:05,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.31 MB 2025-02-15 13:41:05,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47045.41 MB 2025-02-15 13:41:05,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47045.41 MB 2025-02-15 13:41:05,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:41:05,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40869.61 MB 2025-02-15 13:41:05,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:41:05,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:41:05,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.02 seconds 2025-02-15 13:41:05,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28797.15 MB 2025-02-15 13:41:05,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40850.45 MB 2025-02-15 13:41:05,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12053.30 MB 2025-02-15 13:41:05,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56962.84 MB 2025-02-15 13:41:05,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47045.41 MB 2025-02-15 13:41:05,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9917.43 MB 2025-02-15 13:41:05,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40869.61 MB 2025-02-15 13:41:05,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:41:05,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:41:05,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:41:05,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40850.45 MB 2025-02-15 13:41:05,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40950.76 MB 2025-02-15 13:41:05,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.31 MB 2025-02-15 13:41:05,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47045.41 MB 2025-02-15 13:41:05,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47045.41 MB 2025-02-15 13:41:05,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:41:05,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41552.60 MB 2025-02-15 13:41:05,764 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 13:41:05,764 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 13:41:05,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:41:05,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:41:05,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 13:41:05,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:41:05,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30059.85 MB 2025-02-15 13:41:05,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34247.86 MB 2025-02-15 13:41:05,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.01 MB 2025-02-15 13:41:05,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47045.41 MB 2025-02-15 13:41:05,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55421.44 MB 2025-02-15 13:41:05,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 13:41:05,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38435.87 MB 2025-02-15 13:41:05,936 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 13:41:05,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,937 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,938 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:41:05,943 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:41:05,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,944 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:41:05,944 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 13:41:05,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,945 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,945 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:05,951 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:41:05,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,952 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,952 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:05,952 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:41:05,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,953 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:05,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,953 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:41:05,953 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:41:05,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,954 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:05,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,959 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,960 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,961 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:41:05,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:05,970 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:53,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:53,444 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:41:53,452 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:41:53,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:53,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 972, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:41:53,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:41:53,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 972, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:42:08,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:42:08,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:42:08,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.07 seconds 2025-02-15 13:42:08,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:08,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31044.10 MB 2025-02-15 13:42:08,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34483.95 MB 2025-02-15 13:42:08,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3439.85 MB 2025-02-15 13:42:08,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65636.66 MB 2025-02-15 13:42:08,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42502.98 MB 2025-02-15 13:42:08,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23133.68 MB 2025-02-15 13:42:08,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43459.87 MB 2025-02-15 13:42:08,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:42:08,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:42:08,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 13:42:08,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:08,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34483.95 MB 2025-02-15 13:42:08,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32134.36 MB 2025-02-15 13:42:08,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2349.59 MB 2025-02-15 13:42:08,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42502.98 MB 2025-02-15 13:42:08,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49285.17 MB 2025-02-15 13:42:08,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6782.19 MB 2025-02-15 13:42:08,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44932.18 MB 2025-02-15 13:42:10,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:42:10,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:42:10,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:42:10,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32134.36 MB 2025-02-15 13:42:10,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32665.20 MB 2025-02-15 13:42:10,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:42:10,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49285.17 MB 2025-02-15 13:42:10,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 13:42:10,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4657.77 MB 2025-02-15 13:42:10,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36643.75 MB 2025-02-15 13:42:10,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:42:10,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:42:10,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:42:10,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32665.20 MB 2025-02-15 13:42:10,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34554.59 MB 2025-02-15 13:42:10,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.38 MB 2025-02-15 13:42:10,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 13:42:10,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 13:42:10,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:10,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35972.02 MB 2025-02-15 13:42:10,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:42:10,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:42:10,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:42:10,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34554.59 MB 2025-02-15 13:42:10,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36796.44 MB 2025-02-15 13:42:10,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:42:10,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 13:42:10,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 13:42:10,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:10,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42340.72 MB 2025-02-15 13:42:10,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:42:10,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:42:10,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:42:10,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32665.20 MB 2025-02-15 13:42:10,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36796.44 MB 2025-02-15 13:42:10,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.24 MB 2025-02-15 13:42:10,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 13:42:10,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 13:42:10,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:10,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42340.72 MB 2025-02-15 13:42:10,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:42:10,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:42:10,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:42:10,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37504.23 MB 2025-02-15 13:42:10,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38271.23 MB 2025-02-15 13:42:10,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:42:10,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 13:42:10,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45044.73 MB 2025-02-15 13:42:10,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:42:10,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38979.02 MB 2025-02-15 13:42:10,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:42:10,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:42:10,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:42:10,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38684.12 MB 2025-02-15 13:42:10,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38889.65 MB 2025-02-15 13:42:10,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.52 MB 2025-02-15 13:42:10,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45044.73 MB 2025-02-15 13:42:10,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45044.73 MB 2025-02-15 13:42:10,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:10,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39086.72 MB 2025-02-15 13:42:10,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:42:10,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:42:10,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.47 seconds 2025-02-15 13:42:10,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:10,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27657.57 MB 2025-02-15 13:42:10,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39089.78 MB 2025-02-15 13:42:10,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11432.21 MB 2025-02-15 13:42:10,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65636.66 MB 2025-02-15 13:42:10,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45044.73 MB 2025-02-15 13:42:10,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20591.94 MB 2025-02-15 13:42:10,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39089.78 MB 2025-02-15 13:42:11,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:42:11,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:42:11,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:42:11,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:11,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39089.78 MB 2025-02-15 13:42:11,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39189.78 MB 2025-02-15 13:42:11,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.00 MB 2025-02-15 13:42:11,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45044.73 MB 2025-02-15 13:42:11,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45044.73 MB 2025-02-15 13:42:11,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:11,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39789.78 MB 2025-02-15 13:42:11,208 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 13:42:11,209 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:42:11,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:42:11,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:42:11,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:42:11,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:42:11,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28919.56 MB 2025-02-15 13:42:11,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33094.55 MB 2025-02-15 13:42:11,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4174.99 MB 2025-02-15 13:42:11,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45044.73 MB 2025-02-15 13:42:11,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45044.73 MB 2025-02-15 13:42:11,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:42:11,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37269.02 MB 2025-02-15 13:42:11,374 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 13:42:11,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,375 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,376 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:42:11,381 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:42:11,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,382 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:42:11,382 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:42:11,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,382 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,383 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:42:11,389 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:42:11,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,389 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,390 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:42:11,390 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:42:11,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,390 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:42:11,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,391 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:42:11,391 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:42:11,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,391 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:42:11,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,394 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,395 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,395 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,396 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:42:11,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:42:11,405 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:28,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:28,204 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:28,211 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:43:28,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:28,214 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1341, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:43:28,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:28,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1341, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:43:48,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:43:48,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:43:48,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.54 seconds 2025-02-15 13:43:48,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:48,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33736.29 MB 2025-02-15 13:43:48,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38482.01 MB 2025-02-15 13:43:48,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4745.72 MB 2025-02-15 13:43:48,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55381.59 MB 2025-02-15 13:43:48,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52841.94 MB 2025-02-15 13:43:48,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2539.65 MB 2025-02-15 13:43:48,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47284.52 MB 2025-02-15 13:43:48,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:43:48,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:43:48,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:43:48,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:48,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38482.01 MB 2025-02-15 13:43:48,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34172.57 MB 2025-02-15 13:43:48,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4309.44 MB 2025-02-15 13:43:48,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52841.94 MB 2025-02-15 13:43:48,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54257.52 MB 2025-02-15 13:43:48,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 13:43:48,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49547.22 MB 2025-02-15 13:43:50,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:43:50,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:43:50,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-15 13:43:50,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:50,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34172.57 MB 2025-02-15 13:43:50,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34703.41 MB 2025-02-15 13:43:50,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:43:50,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54257.52 MB 2025-02-15 13:43:50,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54257.52 MB 2025-02-15 13:43:50,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:50,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38681.96 MB 2025-02-15 13:43:50,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:43:50,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:43:50,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:43:50,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:50,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34703.41 MB 2025-02-15 13:43:50,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36592.84 MB 2025-02-15 13:43:50,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.43 MB 2025-02-15 13:43:50,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54257.52 MB 2025-02-15 13:43:50,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54257.52 MB 2025-02-15 13:43:50,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:50,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38010.27 MB 2025-02-15 13:43:50,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:43:50,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:43:50,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 13:43:50,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:50,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36592.84 MB 2025-02-15 13:43:50,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38834.70 MB 2025-02-15 13:43:50,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:43:50,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54257.52 MB 2025-02-15 13:43:50,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54257.52 MB 2025-02-15 13:43:50,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:50,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44378.98 MB 2025-02-15 13:43:50,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:43:50,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:43:50,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:43:50,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:50,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34703.41 MB 2025-02-15 13:43:50,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38834.70 MB 2025-02-15 13:43:50,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.29 MB 2025-02-15 13:43:50,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54257.52 MB 2025-02-15 13:43:50,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54257.52 MB 2025-02-15 13:43:50,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:50,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44378.98 MB 2025-02-15 13:43:51,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:43:51,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:43:51,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:43:51,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:51,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39542.49 MB 2025-02-15 13:43:51,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40309.49 MB 2025-02-15 13:43:51,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:43:51,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54257.52 MB 2025-02-15 13:43:51,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54672.75 MB 2025-02-15 13:43:51,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:43:51,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41017.28 MB 2025-02-15 13:43:51,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:43:51,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:43:51,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:43:51,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:51,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40722.38 MB 2025-02-15 13:43:51,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40928.66 MB 2025-02-15 13:43:51,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.29 MB 2025-02-15 13:43:51,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54672.75 MB 2025-02-15 13:43:51,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54672.75 MB 2025-02-15 13:43:51,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:51,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41140.87 MB 2025-02-15 13:43:51,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:43:51,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:43:51,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.92 seconds 2025-02-15 13:43:51,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:51,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29064.14 MB 2025-02-15 13:43:51,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41129.74 MB 2025-02-15 13:43:51,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12065.60 MB 2025-02-15 13:43:51,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55381.59 MB 2025-02-15 13:43:51,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54672.75 MB 2025-02-15 13:43:51,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -708.84 MB 2025-02-15 13:43:51,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41140.87 MB 2025-02-15 13:43:51,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:43:51,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:43:51,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 13:43:51,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:51,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41129.74 MB 2025-02-15 13:43:51,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41230.20 MB 2025-02-15 13:43:51,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:43:51,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54672.75 MB 2025-02-15 13:43:51,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54672.75 MB 2025-02-15 13:43:51,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:51,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41833.00 MB 2025-02-15 13:43:51,427 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:43:51,428 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 13:43:51,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:43:51,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:43:51,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:43:51,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:43:51,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30327.06 MB 2025-02-15 13:43:51,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34521.54 MB 2025-02-15 13:43:51,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:43:51,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54672.75 MB 2025-02-15 13:43:51,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54672.75 MB 2025-02-15 13:43:51,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:43:51,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38715.51 MB 2025-02-15 13:43:51,591 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:43:51,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,592 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,593 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:43:51,598 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:43:51,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,599 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:43:51,599 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 13:43:51,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,600 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,600 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:51,606 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:43:51,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,606 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,607 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:51,607 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:43:51,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,607 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:51,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,608 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:43:51,608 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:43:51,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,608 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:43:51,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,612 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,613 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,614 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:43:51,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:43:51,623 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:44:47,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:44:47,990 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:44:47,995 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:44:47,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:44:47,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1781, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:44:47,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:44:47,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1781, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:45:15,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:45:15,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:45:15,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.53 seconds 2025-02-15 13:45:15,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:15,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36924.66 MB 2025-02-15 13:45:15,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43227.52 MB 2025-02-15 13:45:15,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6302.86 MB 2025-02-15 13:45:15,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65131.25 MB 2025-02-15 13:45:15,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52961.48 MB 2025-02-15 13:45:15,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12169.77 MB 2025-02-15 13:45:15,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52058.34 MB 2025-02-15 13:45:15,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:45:15,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:45:15,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 13:45:15,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:15,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43227.52 MB 2025-02-15 13:45:15,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36582.37 MB 2025-02-15 13:45:15,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6645.14 MB 2025-02-15 13:45:15,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52961.48 MB 2025-02-15 13:45:15,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65219.33 MB 2025-02-15 13:45:15,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12257.85 MB 2025-02-15 13:45:15,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61662.04 MB 2025-02-15 13:45:17,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:45:17,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:45:17,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 13:45:17,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:17,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36582.37 MB 2025-02-15 13:45:17,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37113.21 MB 2025-02-15 13:45:17,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:45:17,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65219.33 MB 2025-02-15 13:45:17,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48767.17 MB 2025-02-15 13:45:17,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16452.16 MB 2025-02-15 13:45:17,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41091.76 MB 2025-02-15 13:45:17,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:45:17,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:45:17,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:45:17,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:17,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37113.21 MB 2025-02-15 13:45:17,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39002.69 MB 2025-02-15 13:45:17,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.48 MB 2025-02-15 13:45:17,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48767.17 MB 2025-02-15 13:45:17,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48767.17 MB 2025-02-15 13:45:17,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:45:17,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40420.12 MB 2025-02-15 13:45:17,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:45:17,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:45:17,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:45:17,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:17,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39002.69 MB 2025-02-15 13:45:17,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41244.55 MB 2025-02-15 13:45:17,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:45:17,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48767.17 MB 2025-02-15 13:45:17,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48767.17 MB 2025-02-15 13:45:17,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:45:17,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46788.83 MB 2025-02-15 13:45:17,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:45:17,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:45:17,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:45:17,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:17,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37113.21 MB 2025-02-15 13:45:17,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41244.55 MB 2025-02-15 13:45:17,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.33 MB 2025-02-15 13:45:17,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48767.17 MB 2025-02-15 13:45:17,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48767.17 MB 2025-02-15 13:45:17,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:45:17,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46788.83 MB 2025-02-15 13:45:18,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:45:18,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:45:18,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:45:18,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:18,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41952.34 MB 2025-02-15 13:45:18,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42719.34 MB 2025-02-15 13:45:18,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:45:18,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48767.17 MB 2025-02-15 13:45:18,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49184.51 MB 2025-02-15 13:45:18,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:45:18,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43427.13 MB 2025-02-15 13:45:18,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:45:18,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:45:18,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:45:18,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:18,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43132.23 MB 2025-02-15 13:45:18,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43337.56 MB 2025-02-15 13:45:18,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.33 MB 2025-02-15 13:45:18,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49184.51 MB 2025-02-15 13:45:18,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49184.51 MB 2025-02-15 13:45:18,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:45:18,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43551.04 MB 2025-02-15 13:45:18,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:45:18,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:45:18,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.10 seconds 2025-02-15 13:45:18,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:18,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30719.51 MB 2025-02-15 13:45:18,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43537.62 MB 2025-02-15 13:45:18,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12818.11 MB 2025-02-15 13:45:18,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65131.25 MB 2025-02-15 13:45:18,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49184.51 MB 2025-02-15 13:45:18,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15946.74 MB 2025-02-15 13:45:18,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43551.04 MB 2025-02-15 13:45:18,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:45:18,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:45:18,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:45:18,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:18,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43537.62 MB 2025-02-15 13:45:18,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43637.58 MB 2025-02-15 13:45:18,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.96 MB 2025-02-15 13:45:18,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49184.51 MB 2025-02-15 13:45:18,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49184.51 MB 2025-02-15 13:45:18,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:45:18,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44237.36 MB 2025-02-15 13:45:18,384 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 13:45:18,385 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:45:18,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:45:18,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:45:18,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:45:18,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:45:18,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31981.43 MB 2025-02-15 13:45:18,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36154.88 MB 2025-02-15 13:45:18,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.45 MB 2025-02-15 13:45:18,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49184.51 MB 2025-02-15 13:45:18,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53357.84 MB 2025-02-15 13:45:18,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-15 13:45:18,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40328.21 MB 2025-02-15 13:45:18,551 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 13:45:18,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,552 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,553 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:45:18,558 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:45:18,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,559 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:45:18,559 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:45:18,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,560 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,560 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:45:18,566 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:45:18,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,567 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,567 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:45:18,567 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:45:18,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,568 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:45:18,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,568 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:45:18,568 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:45:18,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,569 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:45:18,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,575 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,577 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,579 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:45:18,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:45:18,590 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:46:40,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:46:40,119 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:46:40,128 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:46:40,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:46:40,130 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1529, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:46:40,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:46:40,132 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1529, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:47:03,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:47:03,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:47:03,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.81 seconds 2025-02-15 13:47:03,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:03,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35290.34 MB 2025-02-15 13:47:03,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40701.39 MB 2025-02-15 13:47:03,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5411.05 MB 2025-02-15 13:47:03,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63937.97 MB 2025-02-15 13:47:03,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48259.66 MB 2025-02-15 13:47:03,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15678.31 MB 2025-02-15 13:47:03,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49518.05 MB 2025-02-15 13:47:04,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:47:04,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:47:04,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 13:47:04,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:04,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40701.39 MB 2025-02-15 13:47:04,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35393.97 MB 2025-02-15 13:47:04,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5307.42 MB 2025-02-15 13:47:04,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48259.66 MB 2025-02-15 13:47:04,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60305.70 MB 2025-02-15 13:47:04,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12046.04 MB 2025-02-15 13:47:04,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56682.55 MB 2025-02-15 13:47:05,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:47:05,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:47:05,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:47:05,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:05,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35393.97 MB 2025-02-15 13:47:05,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35924.81 MB 2025-02-15 13:47:05,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:47:05,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60305.70 MB 2025-02-15 13:47:05,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44262.49 MB 2025-02-15 13:47:05,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16043.21 MB 2025-02-15 13:47:05,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39903.35 MB 2025-02-15 13:47:06,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:47:06,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:47:06,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:47:06,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35924.81 MB 2025-02-15 13:47:06,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37814.30 MB 2025-02-15 13:47:06,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:47:06,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44262.49 MB 2025-02-15 13:47:06,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44262.49 MB 2025-02-15 13:47:06,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:47:06,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39231.73 MB 2025-02-15 13:47:06,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:47:06,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:47:06,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:47:06,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37814.30 MB 2025-02-15 13:47:06,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40056.16 MB 2025-02-15 13:47:06,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:47:06,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44262.49 MB 2025-02-15 13:47:06,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48037.36 MB 2025-02-15 13:47:06,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 13:47:06,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45600.44 MB 2025-02-15 13:47:06,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:47:06,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:47:06,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:47:06,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35924.81 MB 2025-02-15 13:47:06,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40056.16 MB 2025-02-15 13:47:06,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:47:06,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44262.49 MB 2025-02-15 13:47:06,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48037.36 MB 2025-02-15 13:47:06,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 13:47:06,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45600.44 MB 2025-02-15 13:47:06,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:47:06,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:47:06,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:47:06,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40763.94 MB 2025-02-15 13:47:06,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41530.95 MB 2025-02-15 13:47:06,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:47:06,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48037.36 MB 2025-02-15 13:47:06,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 13:47:06,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:47:06,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42238.73 MB 2025-02-15 13:47:06,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:47:06,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:47:06,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:47:06,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41943.84 MB 2025-02-15 13:47:06,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42149.26 MB 2025-02-15 13:47:06,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.42 MB 2025-02-15 13:47:06,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48454.70 MB 2025-02-15 13:47:06,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 13:47:06,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:47:06,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42367.91 MB 2025-02-15 13:47:06,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:47:06,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:47:06,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.27 seconds 2025-02-15 13:47:06,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29963.18 MB 2025-02-15 13:47:06,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42349.52 MB 2025-02-15 13:47:06,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12386.33 MB 2025-02-15 13:47:06,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63937.97 MB 2025-02-15 13:47:06,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 13:47:06,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15483.27 MB 2025-02-15 13:47:06,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42367.91 MB 2025-02-15 13:47:06,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:47:06,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:47:06,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:47:06,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42349.52 MB 2025-02-15 13:47:06,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42449.58 MB 2025-02-15 13:47:06,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.06 MB 2025-02-15 13:47:06,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48454.70 MB 2025-02-15 13:47:06,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48454.70 MB 2025-02-15 13:47:06,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:47:06,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43049.95 MB 2025-02-15 13:47:06,693 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 13:47:06,693 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:47:06,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:47:06,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:47:06,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:47:06,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:47:06,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31225.29 MB 2025-02-15 13:47:06,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35402.85 MB 2025-02-15 13:47:06,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.56 MB 2025-02-15 13:47:06,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48454.70 MB 2025-02-15 13:47:06,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52632.22 MB 2025-02-15 13:47:06,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-15 13:47:06,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39580.38 MB 2025-02-15 13:47:06,867 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 13:47:06,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,868 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,869 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:47:06,874 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:47:06,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,875 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:47:06,875 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:47:06,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,876 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,876 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:06,882 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:47:06,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,883 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,883 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:06,883 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:47:06,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,884 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:06,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,884 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:47:06,884 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:47:06,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,885 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:06,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,889 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,890 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,892 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:47:06,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:06,902 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:48,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:48,371 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:47:48,376 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:47:48,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:48,377 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1592, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:47:48,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:47:48,378 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1592, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:48:13,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:48:13,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:48:13,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.82 seconds 2025-02-15 13:48:13,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:13,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35850.51 MB 2025-02-15 13:48:13,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41485.56 MB 2025-02-15 13:48:13,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5635.05 MB 2025-02-15 13:48:13,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63331.89 MB 2025-02-15 13:48:13,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53632.57 MB 2025-02-15 13:48:13,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9699.33 MB 2025-02-15 13:48:13,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50304.72 MB 2025-02-15 13:48:13,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:48:13,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:48:13,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:48:13,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:13,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41485.56 MB 2025-02-15 13:48:13,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35842.66 MB 2025-02-15 13:48:13,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5642.90 MB 2025-02-15 13:48:13,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53632.57 MB 2025-02-15 13:48:13,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64520.98 MB 2025-02-15 13:48:13,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10888.41 MB 2025-02-15 13:48:13,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57857.84 MB 2025-02-15 13:48:15,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:48:15,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:48:15,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:48:15,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35842.66 MB 2025-02-15 13:48:15,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36373.50 MB 2025-02-15 13:48:15,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:48:15,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64520.98 MB 2025-02-15 13:48:15,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43824.19 MB 2025-02-15 13:48:15,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20696.79 MB 2025-02-15 13:48:15,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40352.05 MB 2025-02-15 13:48:15,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:48:15,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:48:15,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:48:15,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36373.50 MB 2025-02-15 13:48:15,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38262.99 MB 2025-02-15 13:48:15,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:48:15,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43824.19 MB 2025-02-15 13:48:15,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43824.19 MB 2025-02-15 13:48:15,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:48:15,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39680.42 MB 2025-02-15 13:48:15,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:48:15,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:48:15,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:48:15,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38262.99 MB 2025-02-15 13:48:15,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40504.85 MB 2025-02-15 13:48:15,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:48:15,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43824.19 MB 2025-02-15 13:48:15,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49486.50 MB 2025-02-15 13:48:15,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:48:15,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46049.13 MB 2025-02-15 13:48:15,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:48:15,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:48:15,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:48:15,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36373.50 MB 2025-02-15 13:48:15,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40504.85 MB 2025-02-15 13:48:15,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:48:15,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43824.19 MB 2025-02-15 13:48:15,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49486.50 MB 2025-02-15 13:48:15,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 13:48:15,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46049.13 MB 2025-02-15 13:48:15,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:48:15,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:48:15,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:48:15,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41212.64 MB 2025-02-15 13:48:15,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41979.64 MB 2025-02-15 13:48:15,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:48:15,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49486.50 MB 2025-02-15 13:48:15,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49901.73 MB 2025-02-15 13:48:15,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:48:15,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42687.43 MB 2025-02-15 13:48:15,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:48:15,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:48:15,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:48:15,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42392.53 MB 2025-02-15 13:48:15,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42598.32 MB 2025-02-15 13:48:15,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.79 MB 2025-02-15 13:48:15,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49901.73 MB 2025-02-15 13:48:15,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49901.73 MB 2025-02-15 13:48:15,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:48:15,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42812.71 MB 2025-02-15 13:48:15,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:48:15,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:48:15,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.26 seconds 2025-02-15 13:48:15,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30303.86 MB 2025-02-15 13:48:15,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42798.92 MB 2025-02-15 13:48:15,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12495.06 MB 2025-02-15 13:48:15,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63331.89 MB 2025-02-15 13:48:15,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49901.73 MB 2025-02-15 13:48:15,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13430.16 MB 2025-02-15 13:48:15,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42812.71 MB 2025-02-15 13:48:15,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:48:15,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:48:15,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:48:15,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42798.92 MB 2025-02-15 13:48:15,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42899.15 MB 2025-02-15 13:48:15,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.23 MB 2025-02-15 13:48:15,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49901.73 MB 2025-02-15 13:48:15,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49901.73 MB 2025-02-15 13:48:15,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:48:15,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43500.55 MB 2025-02-15 13:48:15,921 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-15 13:48:15,921 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:48:15,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:48:15,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:48:15,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:48:15,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:48:15,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31566.31 MB 2025-02-15 13:48:15,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35751.05 MB 2025-02-15 13:48:15,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.74 MB 2025-02-15 13:48:15,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49901.73 MB 2025-02-15 13:48:15,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54087.65 MB 2025-02-15 13:48:15,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 13:48:15,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39935.27 MB 2025-02-15 13:48:16,089 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-15 13:48:16,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,092 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:48:16,096 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:48:16,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,097 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:48:16,098 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:48:16,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,098 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,099 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:48:16,105 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:48:16,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,105 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,106 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:48:16,106 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:48:16,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,106 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:48:16,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,107 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:48:16,107 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:48:16,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,107 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:48:16,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,112 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,113 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,114 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:48:16,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:48:16,124 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:04,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:04,151 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:04,156 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:49:04,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:04,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1097, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:49:04,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:04,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1097, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:49:21,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:49:21,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:49:21,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.16 seconds 2025-02-15 13:49:21,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:21,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32523.51 MB 2025-02-15 13:49:21,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36405.73 MB 2025-02-15 13:49:21,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3882.22 MB 2025-02-15 13:49:21,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64908.95 MB 2025-02-15 13:49:21,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43945.82 MB 2025-02-15 13:49:21,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20963.13 MB 2025-02-15 13:49:21,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45392.26 MB 2025-02-15 13:49:21,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:49:21,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:49:21,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 13:49:21,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:21,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36405.73 MB 2025-02-15 13:49:21,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33391.54 MB 2025-02-15 13:49:21,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3014.19 MB 2025-02-15 13:49:21,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43945.82 MB 2025-02-15 13:49:21,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49345.99 MB 2025-02-15 13:49:21,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5400.17 MB 2025-02-15 13:49:21,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45954.53 MB 2025-02-15 13:49:23,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:49:23,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:49:23,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 13:49:23,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33391.54 MB 2025-02-15 13:49:23,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33922.38 MB 2025-02-15 13:49:23,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:49:23,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49345.99 MB 2025-02-15 13:49:23,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-15 13:49:23,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3984.59 MB 2025-02-15 13:49:23,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37900.93 MB 2025-02-15 13:49:23,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:49:23,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:49:23,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:49:23,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33922.38 MB 2025-02-15 13:49:23,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35811.88 MB 2025-02-15 13:49:23,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:49:23,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 13:49:23,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-15 13:49:23,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:49:23,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37229.30 MB 2025-02-15 13:49:23,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:49:23,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:49:23,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 13:49:23,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35811.88 MB 2025-02-15 13:49:23,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38053.73 MB 2025-02-15 13:49:23,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:49:23,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 13:49:23,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-15 13:49:23,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:49:23,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43598.01 MB 2025-02-15 13:49:23,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:49:23,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:49:23,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:49:23,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33922.38 MB 2025-02-15 13:49:23,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38053.73 MB 2025-02-15 13:49:23,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:49:23,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 13:49:23,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-15 13:49:23,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:49:23,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43598.01 MB 2025-02-15 13:49:23,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:49:23,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:49:23,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 13:49:23,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38761.52 MB 2025-02-15 13:49:23,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39528.52 MB 2025-02-15 13:49:23,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:49:23,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-15 13:49:23,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45776.63 MB 2025-02-15 13:49:23,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 13:49:23,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40236.31 MB 2025-02-15 13:49:23,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:49:23,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:49:23,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:49:23,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39941.41 MB 2025-02-15 13:49:23,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40147.29 MB 2025-02-15 13:49:23,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.87 MB 2025-02-15 13:49:23,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45776.63 MB 2025-02-15 13:49:23,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45776.63 MB 2025-02-15 13:49:23,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:49:23,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40357.36 MB 2025-02-15 13:49:23,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:49:23,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:49:23,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.63 seconds 2025-02-15 13:49:23,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:23,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28701.47 MB 2025-02-15 13:49:23,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40348.44 MB 2025-02-15 13:49:23,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11646.97 MB 2025-02-15 13:49:23,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64908.95 MB 2025-02-15 13:49:23,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45776.63 MB 2025-02-15 13:49:23,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19132.32 MB 2025-02-15 13:49:23,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40357.36 MB 2025-02-15 13:49:24,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:49:24,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:49:24,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:49:24,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:24,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40348.44 MB 2025-02-15 13:49:24,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40448.82 MB 2025-02-15 13:49:24,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.38 MB 2025-02-15 13:49:24,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45776.63 MB 2025-02-15 13:49:24,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45776.63 MB 2025-02-15 13:49:24,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:49:24,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41051.11 MB 2025-02-15 13:49:24,076 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 13:49:24,076 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:49:24,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:49:24,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:49:24,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:49:24,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:49:24,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29964.48 MB 2025-02-15 13:49:24,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34155.37 MB 2025-02-15 13:49:24,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.89 MB 2025-02-15 13:49:24,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45776.63 MB 2025-02-15 13:49:24,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49968.84 MB 2025-02-15 13:49:24,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 13:49:24,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38345.75 MB 2025-02-15 13:49:24,239 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 13:49:24,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,241 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:49:24,246 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:49:24,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,247 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:49:24,247 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:49:24,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,248 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,248 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:24,254 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:49:24,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,254 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,255 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:24,255 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:49:24,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,255 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:24,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,256 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:49:24,256 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:49:24,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,256 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:49:24,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,260 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,261 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,262 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:49:24,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:49:24,271 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:18,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:18,077 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:18,082 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:50:18,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:18,084 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1018, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:50:18,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:18,085 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1018, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:50:33,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:50:33,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:50:33,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.76 seconds 2025-02-15 13:50:33,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:33,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32094.73 MB 2025-02-15 13:50:33,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.38 MB 2025-02-15 13:50:33,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3602.64 MB 2025-02-15 13:50:33,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60911.78 MB 2025-02-15 13:50:33,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44065.36 MB 2025-02-15 13:50:33,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16846.42 MB 2025-02-15 13:50:33,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44510.50 MB 2025-02-15 13:50:33,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:50:33,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:50:33,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:50:33,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:33,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35697.38 MB 2025-02-15 13:50:33,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33102.55 MB 2025-02-15 13:50:33,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2594.82 MB 2025-02-15 13:50:33,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44065.36 MB 2025-02-15 13:50:33,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50348.43 MB 2025-02-15 13:50:33,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6283.07 MB 2025-02-15 13:50:33,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46426.27 MB 2025-02-15 13:50:35,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:50:35,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:50:35,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:50:35,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:35,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33102.55 MB 2025-02-15 13:50:35,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33633.39 MB 2025-02-15 13:50:35,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:50:35,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50348.43 MB 2025-02-15 13:50:35,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45480.94 MB 2025-02-15 13:50:35,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4867.49 MB 2025-02-15 13:50:35,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37611.94 MB 2025-02-15 13:50:35,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:50:35,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:50:35,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:50:35,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:35,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33633.39 MB 2025-02-15 13:50:35,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35522.89 MB 2025-02-15 13:50:35,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:50:35,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45480.94 MB 2025-02-15 13:50:35,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45480.94 MB 2025-02-15 13:50:35,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:50:35,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36940.32 MB 2025-02-15 13:50:36,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:50:36,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:50:36,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:50:36,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35522.89 MB 2025-02-15 13:50:36,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37764.74 MB 2025-02-15 13:50:36,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:50:36,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45480.94 MB 2025-02-15 13:50:36,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47368.37 MB 2025-02-15 13:50:36,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:50:36,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43309.03 MB 2025-02-15 13:50:36,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:50:36,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:50:36,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:50:36,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33633.39 MB 2025-02-15 13:50:36,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37764.74 MB 2025-02-15 13:50:36,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:50:36,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45480.94 MB 2025-02-15 13:50:36,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47368.37 MB 2025-02-15 13:50:36,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 13:50:36,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43309.03 MB 2025-02-15 13:50:36,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:50:36,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:50:36,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:50:36,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38472.53 MB 2025-02-15 13:50:36,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39239.53 MB 2025-02-15 13:50:36,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:50:36,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47368.37 MB 2025-02-15 13:50:36,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47785.71 MB 2025-02-15 13:50:36,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:50:36,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39947.32 MB 2025-02-15 13:50:36,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:50:36,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:50:36,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:50:36,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39652.42 MB 2025-02-15 13:50:36,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39860.19 MB 2025-02-15 13:50:36,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.76 MB 2025-02-15 13:50:36,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47785.71 MB 2025-02-15 13:50:36,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47785.71 MB 2025-02-15 13:50:36,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:50:36,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40048.69 MB 2025-02-15 13:50:36,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:50:36,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:50:36,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.16 seconds 2025-02-15 13:50:36,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28547.94 MB 2025-02-15 13:50:36,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40061.26 MB 2025-02-15 13:50:36,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11513.32 MB 2025-02-15 13:50:36,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60911.78 MB 2025-02-15 13:50:36,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47785.71 MB 2025-02-15 13:50:36,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13126.07 MB 2025-02-15 13:50:36,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40061.26 MB 2025-02-15 13:50:36,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:50:36,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:50:36,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:50:36,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40061.26 MB 2025-02-15 13:50:36,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40161.73 MB 2025-02-15 13:50:36,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:50:36,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47785.71 MB 2025-02-15 13:50:36,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47785.71 MB 2025-02-15 13:50:36,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:50:36,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40764.53 MB 2025-02-15 13:50:36,528 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:50:36,529 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:50:36,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:50:36,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:50:36,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:50:36,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:50:36,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29810.86 MB 2025-02-15 13:50:36,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34005.34 MB 2025-02-15 13:50:36,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:50:36,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47785.71 MB 2025-02-15 13:50:36,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51980.01 MB 2025-02-15 13:50:36,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:50:36,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38199.65 MB 2025-02-15 13:50:36,695 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:50:36,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,696 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,697 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:50:36,703 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:50:36,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,704 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:50:36,704 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:50:36,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,706 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,706 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:36,713 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:50:36,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,713 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,714 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:36,714 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:50:36,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,714 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:36,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,715 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:50:36,715 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:50:36,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,716 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:50:36,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,720 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,721 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,722 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:50:36,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:50:36,732 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:51:50,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:51:50,772 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:51:50,777 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:51:50,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:51:50,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:51:50,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:51:50,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:52:09,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:52:09,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:52:09,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.08 seconds 2025-02-15 13:52:09,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:09,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33756.08 MB 2025-02-15 13:52:09,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38140.83 MB 2025-02-15 13:52:09,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4384.75 MB 2025-02-15 13:52:09,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63044.58 MB 2025-02-15 13:52:09,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48381.30 MB 2025-02-15 13:52:09,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14663.29 MB 2025-02-15 13:52:09,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47077.82 MB 2025-02-15 13:52:09,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:52:09,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:52:09,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:52:09,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:09,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38140.83 MB 2025-02-15 13:52:09,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34372.84 MB 2025-02-15 13:52:09,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3767.98 MB 2025-02-15 13:52:09,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48381.30 MB 2025-02-15 13:52:09,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56727.96 MB 2025-02-15 13:52:09,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-15 13:52:09,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50709.49 MB 2025-02-15 13:52:11,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:52:11,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:52:11,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 13:52:11,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:11,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34372.84 MB 2025-02-15 13:52:11,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34903.69 MB 2025-02-15 13:52:11,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:52:11,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56727.96 MB 2025-02-15 13:52:11,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44203.77 MB 2025-02-15 13:52:11,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12524.19 MB 2025-02-15 13:52:11,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38882.23 MB 2025-02-15 13:52:11,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:52:11,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:52:11,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:52:11,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:11,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34903.69 MB 2025-02-15 13:52:11,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36793.18 MB 2025-02-15 13:52:11,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:52:11,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44203.77 MB 2025-02-15 13:52:11,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44203.77 MB 2025-02-15 13:52:11,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:52:11,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38210.61 MB 2025-02-15 13:52:12,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:52:12,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:52:12,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:52:12,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36793.18 MB 2025-02-15 13:52:12,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39035.04 MB 2025-02-15 13:52:12,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:52:12,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44203.77 MB 2025-02-15 13:52:12,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46563.07 MB 2025-02-15 13:52:12,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 13:52:12,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44579.32 MB 2025-02-15 13:52:12,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:52:12,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:52:12,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:52:12,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34903.69 MB 2025-02-15 13:52:12,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39035.04 MB 2025-02-15 13:52:12,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:52:12,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44203.77 MB 2025-02-15 13:52:12,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46563.07 MB 2025-02-15 13:52:12,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 13:52:12,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44579.32 MB 2025-02-15 13:52:12,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:52:12,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:52:12,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:52:12,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39742.82 MB 2025-02-15 13:52:12,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40509.83 MB 2025-02-15 13:52:12,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:52:12,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46563.07 MB 2025-02-15 13:52:12,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46980.40 MB 2025-02-15 13:52:12,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:52:12,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41217.61 MB 2025-02-15 13:52:12,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:52:12,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:52:12,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:52:12,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40922.71 MB 2025-02-15 13:52:12,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41129.41 MB 2025-02-15 13:52:12,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.69 MB 2025-02-15 13:52:12,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46980.40 MB 2025-02-15 13:52:12,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46980.40 MB 2025-02-15 13:52:12,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:52:12,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41338.26 MB 2025-02-15 13:52:12,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:52:12,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:52:12,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.52 seconds 2025-02-15 13:52:12,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29439.30 MB 2025-02-15 13:52:12,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41329.97 MB 2025-02-15 13:52:12,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11890.67 MB 2025-02-15 13:52:12,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63044.58 MB 2025-02-15 13:52:12,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46980.40 MB 2025-02-15 13:52:12,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16064.18 MB 2025-02-15 13:52:12,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41338.26 MB 2025-02-15 13:52:12,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:52:12,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:52:12,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:52:12,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41329.97 MB 2025-02-15 13:52:12,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41430.17 MB 2025-02-15 13:52:12,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.21 MB 2025-02-15 13:52:12,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46980.40 MB 2025-02-15 13:52:12,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46980.40 MB 2025-02-15 13:52:12,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:52:12,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42031.86 MB 2025-02-15 13:52:12,584 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 13:52:12,584 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:52:12,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:52:12,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:52:12,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:52:12,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:52:12,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30701.71 MB 2025-02-15 13:52:12,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34885.42 MB 2025-02-15 13:52:12,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4183.71 MB 2025-02-15 13:52:12,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46980.40 MB 2025-02-15 13:52:12,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51164.22 MB 2025-02-15 13:52:12,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 13:52:12,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39069.24 MB 2025-02-15 13:52:12,747 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 13:52:12,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,749 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,750 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:52:12,754 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:52:12,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,756 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:52:12,756 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:52:12,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,757 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,758 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:52:12,764 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:52:12,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,765 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,766 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:52:12,766 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:52:12,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,766 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:52:12,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,767 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:52:12,767 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:52:12,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,767 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:52:12,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,771 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,772 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,774 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:52:12,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:52:12,784 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:10,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:10,470 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:10,475 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:53:10,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:10,476 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1447, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:53:10,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:10,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1447, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:53:32,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:53:32,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:53:32,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.36 seconds 2025-02-15 13:53:32,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:32,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35327.49 MB 2025-02-15 13:53:32,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40448.34 MB 2025-02-15 13:53:32,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5120.85 MB 2025-02-15 13:53:32,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62350.43 MB 2025-02-15 13:53:32,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48993.67 MB 2025-02-15 13:53:32,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13356.76 MB 2025-02-15 13:53:32,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49328.71 MB 2025-02-15 13:53:32,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:53:32,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:53:32,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 13:53:32,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:32,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40448.34 MB 2025-02-15 13:53:32,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35576.21 MB 2025-02-15 13:53:32,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4872.13 MB 2025-02-15 13:53:32,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48993.67 MB 2025-02-15 13:53:32,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58571.36 MB 2025-02-15 13:53:32,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9577.69 MB 2025-02-15 13:53:32,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54846.43 MB 2025-02-15 13:53:34,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:53:34,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:53:34,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:53:34,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:34,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35576.21 MB 2025-02-15 13:53:34,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36107.05 MB 2025-02-15 13:53:34,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:53:34,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58571.36 MB 2025-02-15 13:53:34,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48993.67 MB 2025-02-15 13:53:34,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9577.69 MB 2025-02-15 13:53:34,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40085.60 MB 2025-02-15 13:53:34,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:53:34,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:53:34,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:53:34,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:34,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36107.05 MB 2025-02-15 13:53:34,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37996.54 MB 2025-02-15 13:53:34,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:53:34,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48993.67 MB 2025-02-15 13:53:34,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48993.67 MB 2025-02-15 13:53:34,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:53:34,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39413.97 MB 2025-02-15 13:53:35,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:53:35,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:53:35,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:53:35,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37996.54 MB 2025-02-15 13:53:35,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40238.40 MB 2025-02-15 13:53:35,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:53:35,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48993.67 MB 2025-02-15 13:53:35,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48993.67 MB 2025-02-15 13:53:35,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:53:35,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45782.68 MB 2025-02-15 13:53:35,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:53:35,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:53:35,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:53:35,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36107.05 MB 2025-02-15 13:53:35,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40238.40 MB 2025-02-15 13:53:35,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:53:35,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48993.67 MB 2025-02-15 13:53:35,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48993.67 MB 2025-02-15 13:53:35,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:53:35,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45782.68 MB 2025-02-15 13:53:35,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:53:35,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:53:35,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:53:35,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40946.19 MB 2025-02-15 13:53:35,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41713.19 MB 2025-02-15 13:53:35,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:53:35,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48993.67 MB 2025-02-15 13:53:35,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49411.00 MB 2025-02-15 13:53:35,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:53:35,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42420.98 MB 2025-02-15 13:53:35,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:53:35,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:53:35,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:53:35,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42126.08 MB 2025-02-15 13:53:35,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42332.06 MB 2025-02-15 13:53:35,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.98 MB 2025-02-15 13:53:35,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49411.00 MB 2025-02-15 13:53:35,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49411.00 MB 2025-02-15 13:53:35,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:53:35,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42538.34 MB 2025-02-15 13:53:35,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:53:35,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:53:35,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.80 seconds 2025-02-15 13:53:35,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30286.03 MB 2025-02-15 13:53:35,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42532.22 MB 2025-02-15 13:53:35,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12246.20 MB 2025-02-15 13:53:35,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62350.43 MB 2025-02-15 13:53:35,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49411.00 MB 2025-02-15 13:53:35,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12939.43 MB 2025-02-15 13:53:35,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42538.34 MB 2025-02-15 13:53:35,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:53:35,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:53:35,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:53:35,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42532.22 MB 2025-02-15 13:53:35,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42632.24 MB 2025-02-15 13:53:35,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.01 MB 2025-02-15 13:53:35,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49411.00 MB 2025-02-15 13:53:35,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49411.00 MB 2025-02-15 13:53:35,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:53:35,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43232.31 MB 2025-02-15 13:53:35,559 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-15 13:53:35,559 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:53:35,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:53:35,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:53:35,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:53:35,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:53:35,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31548.04 MB 2025-02-15 13:53:35,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35723.54 MB 2025-02-15 13:53:35,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.50 MB 2025-02-15 13:53:35,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49411.00 MB 2025-02-15 13:53:35,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53586.43 MB 2025-02-15 13:53:35,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 13:53:35,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39898.97 MB 2025-02-15 13:53:35,727 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-15 13:53:35,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,728 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,729 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:53:35,734 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:53:35,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,735 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:53:35,735 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:53:35,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,736 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,736 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:35,742 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:53:35,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,743 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,743 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:35,743 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:53:35,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,744 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:35,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,744 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:53:35,744 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:53:35,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,745 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:53:35,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,749 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,750 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,752 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:53:35,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:53:35,792 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:54:46,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:54:46,937 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:54:46,942 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:54:46,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:54:46,943 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1236, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:54:46,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:54:46,944 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1236, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:55:05,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:55:05,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:55:05,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.02 seconds 2025-02-15 13:55:05,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:05,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33978.92 MB 2025-02-15 13:55:05,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38353.05 MB 2025-02-15 13:55:05,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4374.13 MB 2025-02-15 13:55:05,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64894.27 MB 2025-02-15 13:55:05,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49098.52 MB 2025-02-15 13:55:05,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15795.75 MB 2025-02-15 13:55:05,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47300.66 MB 2025-02-15 13:55:06,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:55:06,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:55:06,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:55:06,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:06,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38353.05 MB 2025-02-15 13:55:06,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34600.99 MB 2025-02-15 13:55:06,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3752.06 MB 2025-02-15 13:55:06,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49098.52 MB 2025-02-15 13:55:06,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54926.51 MB 2025-02-15 13:55:06,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5827.99 MB 2025-02-15 13:55:06,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50366.99 MB 2025-02-15 13:55:07,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:55:07,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:55:07,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 13:55:07,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:07,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34600.99 MB 2025-02-15 13:55:07,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35131.83 MB 2025-02-15 13:55:07,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:55:07,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54926.51 MB 2025-02-15 13:55:07,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49098.52 MB 2025-02-15 13:55:07,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5827.99 MB 2025-02-15 13:55:07,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39110.38 MB 2025-02-15 13:55:07,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:55:07,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:55:07,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:55:07,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:07,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35131.83 MB 2025-02-15 13:55:07,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37021.33 MB 2025-02-15 13:55:07,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:55:07,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49098.52 MB 2025-02-15 13:55:07,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49098.52 MB 2025-02-15 13:55:07,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:07,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38438.76 MB 2025-02-15 13:55:08,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:55:08,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:55:08,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:55:08,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37021.33 MB 2025-02-15 13:55:08,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39263.18 MB 2025-02-15 13:55:08,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:55:08,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49098.52 MB 2025-02-15 13:55:08,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49098.52 MB 2025-02-15 13:55:08,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:08,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44807.46 MB 2025-02-15 13:55:08,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:55:08,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:55:08,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:55:08,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35131.83 MB 2025-02-15 13:55:08,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39263.18 MB 2025-02-15 13:55:08,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:55:08,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49098.52 MB 2025-02-15 13:55:08,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49098.52 MB 2025-02-15 13:55:08,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:08,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44807.46 MB 2025-02-15 13:55:08,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:55:08,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:55:08,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:55:08,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39970.97 MB 2025-02-15 13:55:08,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40737.97 MB 2025-02-15 13:55:08,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:55:08,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49098.52 MB 2025-02-15 13:55:08,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49515.86 MB 2025-02-15 13:55:08,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:55:08,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41445.76 MB 2025-02-15 13:55:08,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:55:08,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:55:08,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:55:08,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41150.86 MB 2025-02-15 13:55:08,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41357.40 MB 2025-02-15 13:55:08,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.54 MB 2025-02-15 13:55:08,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49515.86 MB 2025-02-15 13:55:08,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49515.86 MB 2025-02-15 13:55:08,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:08,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41556.03 MB 2025-02-15 13:55:08,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:55:08,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:55:08,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.43 seconds 2025-02-15 13:55:08,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29672.59 MB 2025-02-15 13:55:08,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41557.95 MB 2025-02-15 13:55:08,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11885.36 MB 2025-02-15 13:55:08,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64894.27 MB 2025-02-15 13:55:08,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49515.86 MB 2025-02-15 13:55:08,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15378.42 MB 2025-02-15 13:55:08,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41557.95 MB 2025-02-15 13:55:08,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:55:08,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:55:08,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:55:08,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41557.95 MB 2025-02-15 13:55:08,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41658.46 MB 2025-02-15 13:55:08,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.50 MB 2025-02-15 13:55:08,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49515.86 MB 2025-02-15 13:55:08,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49515.86 MB 2025-02-15 13:55:08,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:08,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42259.71 MB 2025-02-15 13:55:08,662 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 13:55:08,662 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:55:08,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:55:08,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:55:08,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:55:08,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:55:08,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30935.29 MB 2025-02-15 13:55:08,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35119.11 MB 2025-02-15 13:55:08,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4183.82 MB 2025-02-15 13:55:08,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49515.86 MB 2025-02-15 13:55:08,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49515.86 MB 2025-02-15 13:55:08,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:55:08,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39302.31 MB 2025-02-15 13:55:08,833 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 13:55:08,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,834 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:55:08,840 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:55:08,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,841 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:55:08,841 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:55:08,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,842 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,842 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:08,848 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:55:08,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,849 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,849 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:08,849 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:55:08,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,850 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:08,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,850 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:55:08,850 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:55:08,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,851 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:08,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,856 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,857 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,858 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:55:08,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:08,870 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:57,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:57,201 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:55:57,206 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:55:57,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:57,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1597, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:55:57,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:55:57,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1597, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:56:21,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:56:21,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:56:21,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.72 seconds 2025-02-15 13:56:21,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:21,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36616.00 MB 2025-02-15 13:56:21,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42267.82 MB 2025-02-15 13:56:21,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5651.82 MB 2025-02-15 13:56:21,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60945.33 MB 2025-02-15 13:56:21,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50686.07 MB 2025-02-15 13:56:21,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10259.27 MB 2025-02-15 13:56:21,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51070.20 MB 2025-02-15 13:56:22,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:56:22,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:56:22,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 13:56:22,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:22,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42267.82 MB 2025-02-15 13:56:22,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36599.30 MB 2025-02-15 13:56:22,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5668.53 MB 2025-02-15 13:56:22,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50686.07 MB 2025-02-15 13:56:22,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61356.38 MB 2025-02-15 13:56:22,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10670.31 MB 2025-02-15 13:56:22,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58313.58 MB 2025-02-15 13:56:23,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:56:23,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:56:23,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:56:23,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:23,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36599.30 MB 2025-02-15 13:56:23,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37130.14 MB 2025-02-15 13:56:23,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:56:23,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61356.38 MB 2025-02-15 13:56:23,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45034.24 MB 2025-02-15 13:56:23,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16322.13 MB 2025-02-15 13:56:23,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41108.69 MB 2025-02-15 13:56:23,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:56:23,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:56:23,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:56:23,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:23,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37130.14 MB 2025-02-15 13:56:23,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39019.63 MB 2025-02-15 13:56:23,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:56:23,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45034.24 MB 2025-02-15 13:56:23,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45034.24 MB 2025-02-15 13:56:23,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:56:23,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40437.06 MB 2025-02-15 13:56:24,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:56:24,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:56:24,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:56:24,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39019.63 MB 2025-02-15 13:56:24,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41261.49 MB 2025-02-15 13:56:24,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:56:24,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45034.24 MB 2025-02-15 13:56:24,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49752.83 MB 2025-02-15 13:56:24,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:56:24,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46805.77 MB 2025-02-15 13:56:24,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:56:24,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:56:24,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:56:24,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37130.14 MB 2025-02-15 13:56:24,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41261.49 MB 2025-02-15 13:56:24,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:56:24,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45034.24 MB 2025-02-15 13:56:24,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49752.83 MB 2025-02-15 13:56:24,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 13:56:24,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46805.77 MB 2025-02-15 13:56:24,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:56:24,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:56:24,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 13:56:24,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41969.28 MB 2025-02-15 13:56:24,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42736.28 MB 2025-02-15 13:56:24,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:56:24,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49752.83 MB 2025-02-15 13:56:24,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50170.17 MB 2025-02-15 13:56:24,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:56:24,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43444.07 MB 2025-02-15 13:56:24,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:56:24,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:56:24,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:56:24,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43149.17 MB 2025-02-15 13:56:24,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43355.64 MB 2025-02-15 13:56:24,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.47 MB 2025-02-15 13:56:24,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50170.17 MB 2025-02-15 13:56:24,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50170.17 MB 2025-02-15 13:56:24,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:56:24,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43551.18 MB 2025-02-15 13:56:24,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:56:24,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:56:24,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.17 seconds 2025-02-15 13:56:24,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31051.92 MB 2025-02-15 13:56:24,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43556.12 MB 2025-02-15 13:56:24,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12504.20 MB 2025-02-15 13:56:24,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60945.33 MB 2025-02-15 13:56:24,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50170.17 MB 2025-02-15 13:56:24,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10775.17 MB 2025-02-15 13:56:24,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43556.12 MB 2025-02-15 13:56:24,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:56:24,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:56:24,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:56:24,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43556.12 MB 2025-02-15 13:56:24,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43656.30 MB 2025-02-15 13:56:24,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.17 MB 2025-02-15 13:56:24,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50170.17 MB 2025-02-15 13:56:24,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50170.17 MB 2025-02-15 13:56:24,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:56:24,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44257.33 MB 2025-02-15 13:56:24,660 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-15 13:56:24,660 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:56:24,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:56:24,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:56:24,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:56:24,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:56:24,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32314.25 MB 2025-02-15 13:56:24,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36496.43 MB 2025-02-15 13:56:24,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4182.17 MB 2025-02-15 13:56:24,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50170.17 MB 2025-02-15 13:56:24,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58535.71 MB 2025-02-15 13:56:24,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-15 13:56:24,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40678.15 MB 2025-02-15 13:56:24,823 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-15 13:56:24,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:56:24,830 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:56:24,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,831 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:56:24,831 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:56:24,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,832 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,832 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:56:24,838 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:56:24,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,839 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,839 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:56:24,839 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:56:24,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,840 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:56:24,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,840 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:56:24,840 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:56:24,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,841 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:56:24,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,845 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,846 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,848 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:56:24,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:56:24,858 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:34,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:34,230 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:34,235 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:57:34,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:34,236 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1213, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:57:34,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:34,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1213, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:57:52,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:57:52,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:57:52,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.70 seconds 2025-02-15 13:57:52,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:52,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34062.07 MB 2025-02-15 13:57:52,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38354.80 MB 2025-02-15 13:57:52,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4292.74 MB 2025-02-15 13:57:52,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70086.82 MB 2025-02-15 13:57:52,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49337.60 MB 2025-02-15 13:57:52,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20749.22 MB 2025-02-15 13:57:52,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47157.32 MB 2025-02-15 13:57:53,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:57:53,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:57:53,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 13:57:53,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:53,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38354.80 MB 2025-02-15 13:57:53,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34724.84 MB 2025-02-15 13:57:53,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3629.96 MB 2025-02-15 13:57:53,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49337.60 MB 2025-02-15 13:57:53,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54676.95 MB 2025-02-15 13:57:53,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5339.35 MB 2025-02-15 13:57:53,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49716.72 MB 2025-02-15 13:57:54,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:57:54,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:57:54,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:57:54,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:54,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34724.84 MB 2025-02-15 13:57:54,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35255.68 MB 2025-02-15 13:57:54,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:57:54,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54676.95 MB 2025-02-15 13:57:54,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49337.60 MB 2025-02-15 13:57:54,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5339.35 MB 2025-02-15 13:57:54,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39234.23 MB 2025-02-15 13:57:54,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:57:54,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:57:54,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:57:54,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:54,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35255.68 MB 2025-02-15 13:57:54,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37145.18 MB 2025-02-15 13:57:54,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:57:54,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49337.60 MB 2025-02-15 13:57:54,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49337.60 MB 2025-02-15 13:57:54,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:57:54,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38562.60 MB 2025-02-15 13:57:55,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:57:55,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:57:55,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:57:55,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37145.18 MB 2025-02-15 13:57:55,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39387.03 MB 2025-02-15 13:57:55,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:57:55,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49337.60 MB 2025-02-15 13:57:55,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49337.60 MB 2025-02-15 13:57:55,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:57:55,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44931.31 MB 2025-02-15 13:57:55,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:57:55,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:57:55,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 13:57:55,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35255.68 MB 2025-02-15 13:57:55,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39387.03 MB 2025-02-15 13:57:55,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:57:55,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49337.60 MB 2025-02-15 13:57:55,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49337.60 MB 2025-02-15 13:57:55,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:57:55,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44931.31 MB 2025-02-15 13:57:55,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:57:55,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:57:55,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 13:57:55,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40094.82 MB 2025-02-15 13:57:55,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40861.82 MB 2025-02-15 13:57:55,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:57:55,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49337.60 MB 2025-02-15 13:57:55,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49754.93 MB 2025-02-15 13:57:55,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 13:57:55,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41569.61 MB 2025-02-15 13:57:55,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:57:55,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:57:55,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:57:55,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41274.71 MB 2025-02-15 13:57:55,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41481.54 MB 2025-02-15 13:57:55,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.83 MB 2025-02-15 13:57:55,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49754.93 MB 2025-02-15 13:57:55,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49754.93 MB 2025-02-15 13:57:55,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:57:55,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41688.21 MB 2025-02-15 13:57:55,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:57:55,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:57:55,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.12 seconds 2025-02-15 13:57:55,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29835.88 MB 2025-02-15 13:57:55,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41682.61 MB 2025-02-15 13:57:55,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11846.74 MB 2025-02-15 13:57:55,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70086.82 MB 2025-02-15 13:57:55,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49754.93 MB 2025-02-15 13:57:55,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20331.89 MB 2025-02-15 13:57:55,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41688.21 MB 2025-02-15 13:57:55,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:57:55,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:57:55,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:57:55,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41682.61 MB 2025-02-15 13:57:55,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41783.08 MB 2025-02-15 13:57:55,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 13:57:55,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49754.93 MB 2025-02-15 13:57:55,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49754.93 MB 2025-02-15 13:57:55,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:57:55,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42385.88 MB 2025-02-15 13:57:55,639 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 13:57:55,639 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:57:55,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:57:55,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:57:55,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:57:55,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:57:55,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31098.80 MB 2025-02-15 13:57:55,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35293.28 MB 2025-02-15 13:57:55,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 13:57:55,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49754.93 MB 2025-02-15 13:57:55,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53949.24 MB 2025-02-15 13:57:55,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 13:57:55,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39487.59 MB 2025-02-15 13:57:55,804 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 13:57:55,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:57:55,811 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:57:55,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,812 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:57:55,812 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 13:57:55,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,813 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,814 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:55,819 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:57:55,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,820 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,820 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:55,820 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:57:55,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,821 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:55,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,821 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:57:55,821 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:57:55,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,822 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:57:55,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,825 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,826 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,827 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:57:55,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:57:55,837 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:58:48,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:58:48,558 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:58:48,563 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 13:58:48,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:58:48,564 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1732, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 13:58:48,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:58:48,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1732, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 13:59:15,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 13:59:15,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 13:59:15,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.70 seconds 2025-02-15 13:59:15,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:15,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37800.25 MB 2025-02-15 13:59:15,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43929.70 MB 2025-02-15 13:59:15,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6129.45 MB 2025-02-15 13:59:15,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65621.98 MB 2025-02-15 13:59:15,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54140.08 MB 2025-02-15 13:59:15,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11481.91 MB 2025-02-15 13:59:15,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52933.93 MB 2025-02-15 13:59:15,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 13:59:15,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 13:59:15,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 13:59:15,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:15,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43929.70 MB 2025-02-15 13:59:15,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37544.67 MB 2025-02-15 13:59:15,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6385.03 MB 2025-02-15 13:59:15,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54140.08 MB 2025-02-15 13:59:15,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66123.20 MB 2025-02-15 13:59:15,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11983.13 MB 2025-02-15 13:59:15,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61949.85 MB 2025-02-15 13:59:17,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 13:59:17,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 13:59:17,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 13:59:17,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37544.67 MB 2025-02-15 13:59:17,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38075.51 MB 2025-02-15 13:59:17,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 13:59:17,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66123.20 MB 2025-02-15 13:59:17,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49964.65 MB 2025-02-15 13:59:17,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16158.56 MB 2025-02-15 13:59:17,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42054.06 MB 2025-02-15 13:59:17,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 13:59:17,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 13:59:17,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:59:17,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38075.51 MB 2025-02-15 13:59:17,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39965.00 MB 2025-02-15 13:59:17,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 13:59:17,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49964.65 MB 2025-02-15 13:59:17,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49964.65 MB 2025-02-15 13:59:17,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:17,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41382.43 MB 2025-02-15 13:59:17,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 13:59:17,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 13:59:17,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:59:17,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39965.00 MB 2025-02-15 13:59:17,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42206.86 MB 2025-02-15 13:59:17,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 13:59:17,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49964.65 MB 2025-02-15 13:59:17,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49964.65 MB 2025-02-15 13:59:17,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:17,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47751.14 MB 2025-02-15 13:59:17,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 13:59:17,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 13:59:17,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 13:59:17,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38075.51 MB 2025-02-15 13:59:17,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42206.86 MB 2025-02-15 13:59:17,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 13:59:17,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49964.65 MB 2025-02-15 13:59:17,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49964.65 MB 2025-02-15 13:59:17,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:17,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47751.14 MB 2025-02-15 13:59:17,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 13:59:17,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 13:59:17,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 13:59:17,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42914.65 MB 2025-02-15 13:59:17,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43681.65 MB 2025-02-15 13:59:17,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 13:59:17,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49964.65 MB 2025-02-15 13:59:17,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50384.08 MB 2025-02-15 13:59:17,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 13:59:17,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44389.44 MB 2025-02-15 13:59:17,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 13:59:17,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 13:59:17,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 13:59:17,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44094.54 MB 2025-02-15 13:59:17,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44300.55 MB 2025-02-15 13:59:17,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.01 MB 2025-02-15 13:59:17,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50384.08 MB 2025-02-15 13:59:17,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50384.08 MB 2025-02-15 13:59:17,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:17,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44520.61 MB 2025-02-15 13:59:17,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 13:59:17,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 13:59:17,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.23 seconds 2025-02-15 13:59:17,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:17,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31765.82 MB 2025-02-15 13:59:17,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44501.40 MB 2025-02-15 13:59:17,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12735.57 MB 2025-02-15 13:59:17,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65621.98 MB 2025-02-15 13:59:17,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50384.08 MB 2025-02-15 13:59:17,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15237.91 MB 2025-02-15 13:59:17,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44520.61 MB 2025-02-15 13:59:18,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 13:59:18,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 13:59:18,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 13:59:18,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:18,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44501.40 MB 2025-02-15 13:59:18,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44601.33 MB 2025-02-15 13:59:18,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.94 MB 2025-02-15 13:59:18,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50384.08 MB 2025-02-15 13:59:18,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50384.08 MB 2025-02-15 13:59:18,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:18,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45200.96 MB 2025-02-15 13:59:18,081 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 13:59:18,081 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:59:18,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 13:59:18,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 13:59:18,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 13:59:18,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 13:59:18,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33027.69 MB 2025-02-15 13:59:18,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37200.11 MB 2025-02-15 13:59:18,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4172.43 MB 2025-02-15 13:59:18,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50384.08 MB 2025-02-15 13:59:18,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50384.08 MB 2025-02-15 13:59:18,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 13:59:18,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41372.02 MB 2025-02-15 13:59:18,244 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 13:59:18,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,246 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,246 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 13:59:18,251 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 13:59:18,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,252 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 13:59:18,252 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 13:59:18,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,253 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,253 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:59:18,259 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 13:59:18,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,260 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,260 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:59:18,260 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 13:59:18,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,261 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:59:18,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,261 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 13:59:18,261 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 13:59:18,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,262 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 13:59:18,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,265 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,266 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,266 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 13:59:18,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 13:59:18,277 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:31,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:31,857 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:31,862 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:00:31,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:31,864 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:00:31,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:31,865 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:00:49,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:00:49,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:00:49,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.98 seconds 2025-02-15 14:00:49,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:49,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33915.33 MB 2025-02-15 14:00:49,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38009.89 MB 2025-02-15 14:00:49,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4094.56 MB 2025-02-15 14:00:49,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62178.46 MB 2025-02-15 14:00:49,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42633.00 MB 2025-02-15 14:00:49,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19545.46 MB 2025-02-15 14:00:49,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47010.58 MB 2025-02-15 14:00:49,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:00:49,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:00:49,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 14:00:49,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:49,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38009.89 MB 2025-02-15 14:00:49,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34678.25 MB 2025-02-15 14:00:49,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3331.64 MB 2025-02-15 14:00:49,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42633.00 MB 2025-02-15 14:00:49,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52380.57 MB 2025-02-15 14:00:49,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9747.56 MB 2025-02-15 14:00:49,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49855.84 MB 2025-02-15 14:00:51,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:00:51,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:00:51,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:00:51,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:51,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34678.25 MB 2025-02-15 14:00:51,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35209.09 MB 2025-02-15 14:00:51,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:00:51,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52380.57 MB 2025-02-15 14:00:51,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-15 14:00:51,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11718.89 MB 2025-02-15 14:00:51,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39187.63 MB 2025-02-15 14:00:51,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:00:51,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:00:51,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:00:51,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:51,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35209.09 MB 2025-02-15 14:00:51,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37098.58 MB 2025-02-15 14:00:51,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:00:51,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-15 14:00:51,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-15 14:00:51,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:00:51,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38516.01 MB 2025-02-15 14:00:52,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:00:52,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:00:52,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 14:00:52,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37098.58 MB 2025-02-15 14:00:52,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39340.44 MB 2025-02-15 14:00:52,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:00:52,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-15 14:00:52,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46795.85 MB 2025-02-15 14:00:52,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 14:00:52,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44884.72 MB 2025-02-15 14:00:52,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:00:52,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:00:52,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:00:52,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35209.09 MB 2025-02-15 14:00:52,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39340.44 MB 2025-02-15 14:00:52,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:00:52,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-15 14:00:52,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46795.85 MB 2025-02-15 14:00:52,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 14:00:52,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44884.72 MB 2025-02-15 14:00:52,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:00:52,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:00:52,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.37 seconds 2025-02-15 14:00:52,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40048.22 MB 2025-02-15 14:00:52,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40815.23 MB 2025-02-15 14:00:52,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:00:52,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46795.85 MB 2025-02-15 14:00:52,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 14:00:52,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:00:52,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41523.02 MB 2025-02-15 14:00:52,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:00:52,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:00:52,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:00:52,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41228.12 MB 2025-02-15 14:00:52,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41436.17 MB 2025-02-15 14:00:52,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.06 MB 2025-02-15 14:00:52,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 14:00:52,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 14:00:52,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:00:52,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41634.72 MB 2025-02-15 14:00:52,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:00:52,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:00:52,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.67 seconds 2025-02-15 14:00:52,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29884.25 MB 2025-02-15 14:00:52,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41637.25 MB 2025-02-15 14:00:52,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11753.00 MB 2025-02-15 14:00:52,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62178.46 MB 2025-02-15 14:00:52,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 14:00:52,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14965.28 MB 2025-02-15 14:00:52,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41637.25 MB 2025-02-15 14:00:52,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:00:52,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:00:52,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 14:00:52,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41637.25 MB 2025-02-15 14:00:52,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41737.71 MB 2025-02-15 14:00:52,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:00:52,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 14:00:52,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47213.18 MB 2025-02-15 14:00:52,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:00:52,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42340.51 MB 2025-02-15 14:00:52,847 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:00:52,847 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:00:52,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:00:52,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:00:52,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:00:52,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:00:52,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31147.17 MB 2025-02-15 14:00:52,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35341.66 MB 2025-02-15 14:00:52,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:00:52,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47213.18 MB 2025-02-15 14:00:52,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55603.89 MB 2025-02-15 14:00:52,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 14:00:52,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39535.96 MB 2025-02-15 14:00:53,108 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:00:53,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,112 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:00:53,119 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:00:53,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,121 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:00:53,122 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:00:53,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,123 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,124 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:53,134 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:00:53,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,135 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,136 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:53,136 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:00:53,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,137 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:53,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,138 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:00:53,138 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:00:53,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,139 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:00:53,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,145 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,147 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,149 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:00:53,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:00:53,160 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:14,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:14,563 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:14,568 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:02:14,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:14,569 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1580, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:02:14,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:14,570 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1580, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:02:39,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:02:39,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:02:39,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.45 seconds 2025-02-15 14:02:39,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:39,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36984.64 MB 2025-02-15 14:02:39,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42576.17 MB 2025-02-15 14:02:39,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5591.53 MB 2025-02-15 14:02:39,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67519.91 MB 2025-02-15 14:02:39,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52640.61 MB 2025-02-15 14:02:39,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14879.29 MB 2025-02-15 14:02:39,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51438.84 MB 2025-02-15 14:02:39,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:02:39,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:02:39,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 14:02:39,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:39,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42576.17 MB 2025-02-15 14:02:39,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36998.02 MB 2025-02-15 14:02:39,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5578.15 MB 2025-02-15 14:02:39,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52640.61 MB 2025-02-15 14:02:39,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63529.03 MB 2025-02-15 14:02:39,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10888.41 MB 2025-02-15 14:02:39,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58949.77 MB 2025-02-15 14:02:41,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:02:41,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:02:41,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:02:41,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36998.02 MB 2025-02-15 14:02:41,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37528.86 MB 2025-02-15 14:02:41,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:02:41,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63529.03 MB 2025-02-15 14:02:41,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47047.51 MB 2025-02-15 14:02:41,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16481.52 MB 2025-02-15 14:02:41,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41507.40 MB 2025-02-15 14:02:41,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:02:41,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:02:41,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:02:41,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37528.86 MB 2025-02-15 14:02:41,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39418.35 MB 2025-02-15 14:02:41,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:02:41,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47047.51 MB 2025-02-15 14:02:41,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47047.51 MB 2025-02-15 14:02:41,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:02:41,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40835.78 MB 2025-02-15 14:02:41,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:02:41,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:02:41,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:02:41,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39418.35 MB 2025-02-15 14:02:41,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41660.21 MB 2025-02-15 14:02:41,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:02:41,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47047.51 MB 2025-02-15 14:02:41,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49406.80 MB 2025-02-15 14:02:41,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 14:02:41,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47204.49 MB 2025-02-15 14:02:41,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:02:41,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:02:41,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:02:41,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37528.86 MB 2025-02-15 14:02:41,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41660.21 MB 2025-02-15 14:02:41,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:02:41,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47047.51 MB 2025-02-15 14:02:41,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49406.80 MB 2025-02-15 14:02:41,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 14:02:41,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47204.49 MB 2025-02-15 14:02:41,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:02:41,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:02:41,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:02:41,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42368.00 MB 2025-02-15 14:02:41,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43135.00 MB 2025-02-15 14:02:41,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:02:41,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49406.80 MB 2025-02-15 14:02:41,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49824.14 MB 2025-02-15 14:02:41,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:02:41,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43842.79 MB 2025-02-15 14:02:41,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:02:41,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:02:41,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:02:41,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43547.89 MB 2025-02-15 14:02:41,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43754.45 MB 2025-02-15 14:02:41,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.56 MB 2025-02-15 14:02:41,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49824.14 MB 2025-02-15 14:02:41,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49824.14 MB 2025-02-15 14:02:41,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:02:41,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43952.73 MB 2025-02-15 14:02:41,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:02:41,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:02:41,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.08 seconds 2025-02-15 14:02:41,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31479.79 MB 2025-02-15 14:02:41,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43955.30 MB 2025-02-15 14:02:41,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12475.51 MB 2025-02-15 14:02:41,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67519.91 MB 2025-02-15 14:02:41,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49824.14 MB 2025-02-15 14:02:41,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17695.77 MB 2025-02-15 14:02:41,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43955.30 MB 2025-02-15 14:02:41,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:02:41,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:02:41,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:02:41,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43955.30 MB 2025-02-15 14:02:41,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44055.59 MB 2025-02-15 14:02:41,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.29 MB 2025-02-15 14:02:41,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49824.14 MB 2025-02-15 14:02:41,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49824.14 MB 2025-02-15 14:02:41,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:02:41,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44657.59 MB 2025-02-15 14:02:41,932 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 14:02:41,932 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:02:41,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:02:41,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:02:41,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:02:41,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:02:41,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32742.37 MB 2025-02-15 14:02:41,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36929.67 MB 2025-02-15 14:02:41,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4187.30 MB 2025-02-15 14:02:41,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49824.14 MB 2025-02-15 14:02:41,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54012.15 MB 2025-02-15 14:02:41,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 14:02:41,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41116.46 MB 2025-02-15 14:02:42,096 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 14:02:42,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,098 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,098 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:02:42,103 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:02:42,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,104 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:02:42,104 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:02:42,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,105 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,105 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:42,111 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:02:42,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,112 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,112 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:42,112 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:02:42,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,113 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:42,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,113 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:02:42,113 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:02:42,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,114 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:02:42,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,117 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,118 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,119 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:02:42,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:02:42,130 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:30,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:30,706 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:30,711 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:03:30,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:30,712 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1675, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:03:30,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:30,713 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1675, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:03:56,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:03:56,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:03:56,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.16 seconds 2025-02-15 14:03:56,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:56,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37768.39 MB 2025-02-15 14:03:56,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43697.04 MB 2025-02-15 14:03:56,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5928.65 MB 2025-02-15 14:03:56,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66049.80 MB 2025-02-15 14:03:56,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53091.50 MB 2025-02-15 14:03:56,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12958.30 MB 2025-02-15 14:03:56,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52675.58 MB 2025-02-15 14:03:56,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:03:56,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:03:56,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 14:03:56,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:56,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43697.04 MB 2025-02-15 14:03:56,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37613.67 MB 2025-02-15 14:03:56,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6083.37 MB 2025-02-15 14:03:56,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53091.50 MB 2025-02-15 14:03:56,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64567.12 MB 2025-02-15 14:03:56,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11475.62 MB 2025-02-15 14:03:56,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58825.99 MB 2025-02-15 14:03:58,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:03:58,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:03:58,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:03:58,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:58,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37613.67 MB 2025-02-15 14:03:58,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38144.51 MB 2025-02-15 14:03:58,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:03:58,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64567.12 MB 2025-02-15 14:03:58,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48578.43 MB 2025-02-15 14:03:58,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15988.69 MB 2025-02-15 14:03:58,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42123.05 MB 2025-02-15 14:03:58,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:03:58,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:03:58,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:03:58,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:58,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38144.51 MB 2025-02-15 14:03:58,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40034.00 MB 2025-02-15 14:03:58,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:03:58,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48578.43 MB 2025-02-15 14:03:58,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48578.43 MB 2025-02-15 14:03:58,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:03:58,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41451.43 MB 2025-02-15 14:03:59,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:03:59,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:03:59,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:03:59,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40034.00 MB 2025-02-15 14:03:59,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42275.86 MB 2025-02-15 14:03:59,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:03:59,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48578.43 MB 2025-02-15 14:03:59,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50465.87 MB 2025-02-15 14:03:59,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 14:03:59,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47820.14 MB 2025-02-15 14:03:59,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:03:59,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:03:59,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:03:59,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38144.51 MB 2025-02-15 14:03:59,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42275.86 MB 2025-02-15 14:03:59,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:03:59,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48578.43 MB 2025-02-15 14:03:59,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50465.87 MB 2025-02-15 14:03:59,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 14:03:59,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47820.14 MB 2025-02-15 14:03:59,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:03:59,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:03:59,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:03:59,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42983.64 MB 2025-02-15 14:03:59,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43750.65 MB 2025-02-15 14:03:59,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:03:59,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50465.87 MB 2025-02-15 14:03:59,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50883.20 MB 2025-02-15 14:03:59,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:03:59,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44458.44 MB 2025-02-15 14:03:59,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:03:59,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:03:59,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:03:59,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44163.54 MB 2025-02-15 14:03:59,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44371.13 MB 2025-02-15 14:03:59,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.59 MB 2025-02-15 14:03:59,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-15 14:03:59,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50883.20 MB 2025-02-15 14:03:59,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:03:59,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44573.16 MB 2025-02-15 14:03:59,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:03:59,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:03:59,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.61 seconds 2025-02-15 14:03:59,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31932.55 MB 2025-02-15 14:03:59,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44572.20 MB 2025-02-15 14:03:59,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12639.65 MB 2025-02-15 14:03:59,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66049.80 MB 2025-02-15 14:03:59,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50883.20 MB 2025-02-15 14:03:59,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15166.60 MB 2025-02-15 14:03:59,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44573.16 MB 2025-02-15 14:03:59,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:03:59,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:03:59,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:03:59,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44572.20 MB 2025-02-15 14:03:59,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44672.67 MB 2025-02-15 14:03:59,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:03:59,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-15 14:03:59,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50883.20 MB 2025-02-15 14:03:59,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:03:59,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45275.47 MB 2025-02-15 14:03:59,638 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:03:59,639 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:03:59,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:03:59,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:03:59,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 14:03:59,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:03:59,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33195.47 MB 2025-02-15 14:03:59,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37389.96 MB 2025-02-15 14:03:59,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:03:59,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-15 14:03:59,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59273.90 MB 2025-02-15 14:03:59,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 14:03:59,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41584.26 MB 2025-02-15 14:03:59,814 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:03:59,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,816 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:03:59,821 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:03:59,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,822 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:03:59,823 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:03:59,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,823 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,824 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:59,830 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:03:59,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,830 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,831 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:59,831 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:03:59,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,831 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:59,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,832 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:03:59,832 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:03:59,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,832 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:03:59,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,838 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,840 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,842 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:03:59,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:03:59,853 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:29,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:29,133 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:29,138 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:05:29,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:29,140 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1325, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:05:29,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:29,141 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1325, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:05:49,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:05:49,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:05:49,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.43 seconds 2025-02-15 14:05:49,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:49,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35451.30 MB 2025-02-15 14:05:49,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40140.54 MB 2025-02-15 14:05:49,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4689.23 MB 2025-02-15 14:05:49,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71433.19 MB 2025-02-15 14:05:49,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51971.62 MB 2025-02-15 14:05:49,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19461.57 MB 2025-02-15 14:05:49,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48999.54 MB 2025-02-15 14:05:49,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:05:49,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:05:49,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 14:05:49,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:49,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40140.54 MB 2025-02-15 14:05:49,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35915.90 MB 2025-02-15 14:05:49,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4224.64 MB 2025-02-15 14:05:49,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51971.62 MB 2025-02-15 14:05:49,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55301.90 MB 2025-02-15 14:05:49,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3330.28 MB 2025-02-15 14:05:49,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50974.12 MB 2025-02-15 14:05:51,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:05:51,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:05:51,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:05:51,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35915.90 MB 2025-02-15 14:05:51,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36446.74 MB 2025-02-15 14:05:51,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:05:51,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55301.90 MB 2025-02-15 14:05:51,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-15 14:05:51,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8019.51 MB 2025-02-15 14:05:51,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40425.29 MB 2025-02-15 14:05:51,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:05:51,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:05:51,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:05:51,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36446.74 MB 2025-02-15 14:05:51,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38336.23 MB 2025-02-15 14:05:51,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:05:51,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-15 14:05:51,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-15 14:05:51,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:05:51,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39753.66 MB 2025-02-15 14:05:51,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:05:51,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:05:51,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:05:51,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38336.23 MB 2025-02-15 14:05:51,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40578.09 MB 2025-02-15 14:05:51,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:05:51,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-15 14:05:51,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48700.06 MB 2025-02-15 14:05:51,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-15 14:05:51,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46122.37 MB 2025-02-15 14:05:51,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:05:51,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:05:51,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:05:51,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36446.74 MB 2025-02-15 14:05:51,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40578.09 MB 2025-02-15 14:05:51,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:05:51,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-15 14:05:51,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48700.06 MB 2025-02-15 14:05:51,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-15 14:05:51,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46122.37 MB 2025-02-15 14:05:51,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:05:51,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:05:51,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:05:51,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41285.88 MB 2025-02-15 14:05:51,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42052.88 MB 2025-02-15 14:05:51,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:05:51,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48700.06 MB 2025-02-15 14:05:51,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-15 14:05:51,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:05:51,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42760.67 MB 2025-02-15 14:05:51,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:05:51,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:05:51,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:05:51,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42465.77 MB 2025-02-15 14:05:51,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42673.02 MB 2025-02-15 14:05:51,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.25 MB 2025-02-15 14:05:51,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49117.40 MB 2025-02-15 14:05:51,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-15 14:05:51,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:05:51,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42877.05 MB 2025-02-15 14:05:51,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:05:51,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:05:51,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.83 seconds 2025-02-15 14:05:51,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:51,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30834.90 MB 2025-02-15 14:05:51,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42874.09 MB 2025-02-15 14:05:51,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12039.19 MB 2025-02-15 14:05:51,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71433.19 MB 2025-02-15 14:05:51,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-15 14:05:51,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22315.79 MB 2025-02-15 14:05:51,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42877.05 MB 2025-02-15 14:05:52,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:05:52,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:05:52,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:05:52,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:52,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42874.09 MB 2025-02-15 14:05:52,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42974.56 MB 2025-02-15 14:05:52,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:05:52,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49117.40 MB 2025-02-15 14:05:52,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-15 14:05:52,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:05:52,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43577.36 MB 2025-02-15 14:05:52,260 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:05:52,261 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 14:05:52,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:05:52,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:05:52,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:05:52,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:05:52,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32097.82 MB 2025-02-15 14:05:52,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36292.30 MB 2025-02-15 14:05:52,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:05:52,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49117.40 MB 2025-02-15 14:05:52,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57508.10 MB 2025-02-15 14:05:52,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 14:05:52,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40486.61 MB 2025-02-15 14:05:52,427 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:05:52,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,429 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,430 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:05:52,434 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:05:52,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,435 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:05:52,435 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 14:05:52,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,436 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,437 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:52,443 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:05:52,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,443 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,444 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:52,444 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:05:52,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,444 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:52,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,445 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:05:52,445 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:05:52,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,445 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:05:52,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,452 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,454 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,456 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:05:52,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:05:52,468 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:06:46,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:06:46,586 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:06:46,591 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:06:46,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:06:46,592 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2139, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:06:46,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:06:46,593 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2139, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:07:20,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:07:20,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:07:20,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.43 seconds 2025-02-15 14:07:20,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:20,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41245.16 MB 2025-02-15 14:07:20,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48815.88 MB 2025-02-15 14:07:20,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7570.72 MB 2025-02-15 14:07:20,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69789.02 MB 2025-02-15 14:07:20,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54974.74 MB 2025-02-15 14:07:20,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14814.28 MB 2025-02-15 14:07:20,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57738.43 MB 2025-02-15 14:07:20,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:07:20,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:07:20,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 14:07:20,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:20,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48815.88 MB 2025-02-15 14:07:20,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40270.45 MB 2025-02-15 14:07:20,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8545.43 MB 2025-02-15 14:07:20,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54974.74 MB 2025-02-15 14:07:20,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71787.61 MB 2025-02-15 14:07:20,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16812.87 MB 2025-02-15 14:07:20,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70905.67 MB 2025-02-15 14:07:22,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:07:22,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:07:22,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 14:07:22,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40270.45 MB 2025-02-15 14:07:22,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40801.29 MB 2025-02-15 14:07:22,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:07:22,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71787.61 MB 2025-02-15 14:07:22,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45334.13 MB 2025-02-15 14:07:22,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26453.48 MB 2025-02-15 14:07:22,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44780.88 MB 2025-02-15 14:07:22,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:07:22,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:07:22,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:07:22,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40801.29 MB 2025-02-15 14:07:22,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42690.79 MB 2025-02-15 14:07:22,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:07:22,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45334.13 MB 2025-02-15 14:07:22,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47221.57 MB 2025-02-15 14:07:22,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 14:07:22,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44108.22 MB 2025-02-15 14:07:22,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:07:22,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:07:22,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:07:22,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42690.79 MB 2025-02-15 14:07:22,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44932.64 MB 2025-02-15 14:07:22,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:07:22,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47221.57 MB 2025-02-15 14:07:22,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52885.98 MB 2025-02-15 14:07:22,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 14:07:22,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50476.92 MB 2025-02-15 14:07:22,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:07:22,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:07:22,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:07:22,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40801.29 MB 2025-02-15 14:07:22,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44932.64 MB 2025-02-15 14:07:22,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:07:22,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45334.13 MB 2025-02-15 14:07:22,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52885.98 MB 2025-02-15 14:07:22,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-15 14:07:22,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50476.92 MB 2025-02-15 14:07:22,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:07:22,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:07:22,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:07:22,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45640.43 MB 2025-02-15 14:07:22,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46407.43 MB 2025-02-15 14:07:22,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:07:22,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52885.98 MB 2025-02-15 14:07:22,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53303.31 MB 2025-02-15 14:07:22,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:07:22,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47115.22 MB 2025-02-15 14:07:22,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:07:22,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:07:22,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:07:22,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46820.32 MB 2025-02-15 14:07:22,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47027.26 MB 2025-02-15 14:07:22,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.94 MB 2025-02-15 14:07:22,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53303.31 MB 2025-02-15 14:07:22,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53303.31 MB 2025-02-15 14:07:22,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:07:22,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47228.10 MB 2025-02-15 14:07:22,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:07:22,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:07:22,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.01 seconds 2025-02-15 14:07:22,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33792.71 MB 2025-02-15 14:07:22,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47228.14 MB 2025-02-15 14:07:22,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13435.43 MB 2025-02-15 14:07:22,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69789.02 MB 2025-02-15 14:07:22,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53303.31 MB 2025-02-15 14:07:22,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16485.71 MB 2025-02-15 14:07:22,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47228.14 MB 2025-02-15 14:07:22,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:07:22,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:07:22,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:07:22,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47228.14 MB 2025-02-15 14:07:22,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47328.51 MB 2025-02-15 14:07:22,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 14:07:22,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53303.31 MB 2025-02-15 14:07:22,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53303.31 MB 2025-02-15 14:07:22,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:07:22,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47930.72 MB 2025-02-15 14:07:22,888 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 14:07:22,889 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:07:22,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:07:22,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:07:22,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:07:22,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:07:22,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35055.44 MB 2025-02-15 14:07:22,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39245.82 MB 2025-02-15 14:07:22,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 14:07:22,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53303.31 MB 2025-02-15 14:07:22,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57493.42 MB 2025-02-15 14:07:22,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 14:07:22,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43435.93 MB 2025-02-15 14:07:23,053 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 14:07:23,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:07:23,060 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:07:23,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,061 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:07:23,061 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:07:23,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,062 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,063 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:07:23,068 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:07:23,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,069 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,070 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:07:23,070 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:07:23,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,070 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:07:23,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,071 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:07:23,071 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:07:23,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,071 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:07:23,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,076 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,078 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,079 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:07:23,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:07:23,090 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:03,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:03,033 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:03,038 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:09:03,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:03,039 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1125, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:09:03,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:03,040 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1125, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:09:20,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:09:20,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:09:20,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.46 seconds 2025-02-15 14:09:20,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:20,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34301.22 MB 2025-02-15 14:09:20,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38282.53 MB 2025-02-15 14:09:20,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3981.31 MB 2025-02-15 14:09:20,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69895.98 MB 2025-02-15 14:09:20,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43123.74 MB 2025-02-15 14:09:20,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26772.24 MB 2025-02-15 14:09:20,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47169.98 MB 2025-02-15 14:09:20,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:09:20,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:09:20,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 14:09:20,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:20,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38282.53 MB 2025-02-15 14:09:20,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35120.76 MB 2025-02-15 14:09:20,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3161.78 MB 2025-02-15 14:09:20,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43123.74 MB 2025-02-15 14:09:20,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52336.53 MB 2025-02-15 14:09:20,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9212.79 MB 2025-02-15 14:09:20,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49595.36 MB 2025-02-15 14:09:22,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:09:22,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:09:22,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:09:22,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:22,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35120.76 MB 2025-02-15 14:09:22,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35651.60 MB 2025-02-15 14:09:22,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:09:22,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52336.53 MB 2025-02-15 14:09:22,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41265.66 MB 2025-02-15 14:09:22,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11070.87 MB 2025-02-15 14:09:22,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39630.15 MB 2025-02-15 14:09:22,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:09:22,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:09:22,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:09:22,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:22,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35651.60 MB 2025-02-15 14:09:22,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37541.09 MB 2025-02-15 14:09:22,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:09:22,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41265.66 MB 2025-02-15 14:09:22,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41265.66 MB 2025-02-15 14:09:22,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:09:22,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38958.52 MB 2025-02-15 14:09:22,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:09:22,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:09:22,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 14:09:22,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:22,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37541.09 MB 2025-02-15 14:09:22,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39782.95 MB 2025-02-15 14:09:22,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:09:22,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41265.66 MB 2025-02-15 14:09:22,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47399.83 MB 2025-02-15 14:09:22,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 14:09:22,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45327.23 MB 2025-02-15 14:09:22,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:09:22,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:09:22,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 14:09:22,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:22,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35651.60 MB 2025-02-15 14:09:22,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39782.95 MB 2025-02-15 14:09:22,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:09:22,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41265.66 MB 2025-02-15 14:09:22,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47399.83 MB 2025-02-15 14:09:22,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 14:09:22,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45327.23 MB 2025-02-15 14:09:23,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:09:23,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:09:23,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:09:23,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:23,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40490.74 MB 2025-02-15 14:09:23,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41257.74 MB 2025-02-15 14:09:23,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:09:23,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47399.83 MB 2025-02-15 14:09:23,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47817.16 MB 2025-02-15 14:09:23,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:09:23,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41965.53 MB 2025-02-15 14:09:23,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:09:23,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:09:23,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:09:23,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:23,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41670.63 MB 2025-02-15 14:09:23,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41875.98 MB 2025-02-15 14:09:23,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.36 MB 2025-02-15 14:09:23,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47817.16 MB 2025-02-15 14:09:23,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47817.16 MB 2025-02-15 14:09:23,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:09:23,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42067.26 MB 2025-02-15 14:09:23,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:09:23,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:09:23,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.05 seconds 2025-02-15 14:09:23,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:23,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30381.63 MB 2025-02-15 14:09:23,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42076.22 MB 2025-02-15 14:09:23,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11694.59 MB 2025-02-15 14:09:23,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69895.98 MB 2025-02-15 14:09:23,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47817.16 MB 2025-02-15 14:09:23,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22078.82 MB 2025-02-15 14:09:23,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42076.22 MB 2025-02-15 14:09:23,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:09:23,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:09:23,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:09:23,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:23,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42076.22 MB 2025-02-15 14:09:23,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42176.27 MB 2025-02-15 14:09:23,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.05 MB 2025-02-15 14:09:23,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47817.16 MB 2025-02-15 14:09:23,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47817.16 MB 2025-02-15 14:09:23,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:09:23,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42776.56 MB 2025-02-15 14:09:23,383 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 14:09:23,384 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:09:23,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:09:23,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:09:23,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:09:23,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:09:23,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31643.72 MB 2025-02-15 14:09:23,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35821.24 MB 2025-02-15 14:09:23,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.53 MB 2025-02-15 14:09:23,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47817.16 MB 2025-02-15 14:09:23,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56172.22 MB 2025-02-15 14:09:23,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 14:09:23,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39998.77 MB 2025-02-15 14:09:23,576 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 14:09:23,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,577 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,578 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:09:23,583 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:09:23,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,584 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:09:23,584 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:09:23,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,585 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,586 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:23,592 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:09:23,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,593 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,593 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:23,593 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:09:23,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,594 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:23,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,594 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:09:23,595 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:09:23,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,595 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:09:23,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,602 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,604 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,607 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:09:23,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:09:23,619 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:10:29,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:10:29,386 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:10:29,392 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:10:29,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:10:29,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2392, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:10:29,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:10:29,394 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2392, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:11:06,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:11:06,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:11:06,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.10 seconds 2025-02-15 14:11:06,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:06,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43251.65 MB 2025-02-15 14:11:06,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51717.86 MB 2025-02-15 14:11:06,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8466.20 MB 2025-02-15 14:11:06,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68696.41 MB 2025-02-15 14:11:06,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60112.76 MB 2025-02-15 14:11:06,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8583.64 MB 2025-02-15 14:11:06,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60650.26 MB 2025-02-15 14:11:06,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:11:06,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:11:06,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 14:11:06,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:06,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51717.86 MB 2025-02-15 14:11:06,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41828.22 MB 2025-02-15 14:11:06,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9889.64 MB 2025-02-15 14:11:06,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60112.76 MB 2025-02-15 14:11:06,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76759.96 MB 2025-02-15 14:11:06,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16647.19 MB 2025-02-15 14:11:06,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 76753.04 MB 2025-02-15 14:11:08,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:11:08,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:11:08,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-15 14:11:08,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:08,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41828.22 MB 2025-02-15 14:11:08,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42359.06 MB 2025-02-15 14:11:08,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:11:08,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76759.96 MB 2025-02-15 14:11:08,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51646.56 MB 2025-02-15 14:11:08,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25113.40 MB 2025-02-15 14:11:08,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46337.61 MB 2025-02-15 14:11:08,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:11:08,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:11:08,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:11:08,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:08,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42359.06 MB 2025-02-15 14:11:08,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44248.56 MB 2025-02-15 14:11:08,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:11:08,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51646.56 MB 2025-02-15 14:11:08,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51646.56 MB 2025-02-15 14:11:08,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:11:08,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45665.98 MB 2025-02-15 14:11:08,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:11:08,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:11:08,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:11:08,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:08,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44248.56 MB 2025-02-15 14:11:08,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46490.41 MB 2025-02-15 14:11:08,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:11:08,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51646.56 MB 2025-02-15 14:11:08,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54949.58 MB 2025-02-15 14:11:08,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 14:11:08,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52034.69 MB 2025-02-15 14:11:08,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:11:08,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:11:08,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:11:08,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:08,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42359.06 MB 2025-02-15 14:11:08,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46490.41 MB 2025-02-15 14:11:08,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:11:08,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51646.56 MB 2025-02-15 14:11:08,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54949.58 MB 2025-02-15 14:11:08,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 14:11:08,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52034.69 MB 2025-02-15 14:11:09,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:11:09,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:11:09,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:11:09,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:09,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47198.20 MB 2025-02-15 14:11:09,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47965.20 MB 2025-02-15 14:11:09,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:11:09,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54949.58 MB 2025-02-15 14:11:09,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55366.91 MB 2025-02-15 14:11:09,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:11:09,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48672.99 MB 2025-02-15 14:11:09,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:11:09,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:11:09,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:11:09,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:09,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48378.09 MB 2025-02-15 14:11:09,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48584.59 MB 2025-02-15 14:11:09,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.50 MB 2025-02-15 14:11:09,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55366.91 MB 2025-02-15 14:11:09,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55366.91 MB 2025-02-15 14:11:09,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:11:09,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48816.75 MB 2025-02-15 14:11:09,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:11:09,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:11:09,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.72 seconds 2025-02-15 14:11:09,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:09,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34917.73 MB 2025-02-15 14:11:09,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48785.44 MB 2025-02-15 14:11:09,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13867.71 MB 2025-02-15 14:11:09,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68696.41 MB 2025-02-15 14:11:09,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55366.91 MB 2025-02-15 14:11:09,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13329.50 MB 2025-02-15 14:11:09,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48816.75 MB 2025-02-15 14:11:09,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:11:09,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:11:09,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:11:09,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:09,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48785.44 MB 2025-02-15 14:11:09,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48885.65 MB 2025-02-15 14:11:09,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.21 MB 2025-02-15 14:11:09,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55366.91 MB 2025-02-15 14:11:09,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55366.91 MB 2025-02-15 14:11:09,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:11:09,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49486.90 MB 2025-02-15 14:11:09,397 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 14:11:09,397 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 14:11:09,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:11:09,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:11:09,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:11:09,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:11:09,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36180.14 MB 2025-02-15 14:11:09,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40363.85 MB 2025-02-15 14:11:09,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4183.71 MB 2025-02-15 14:11:09,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55366.91 MB 2025-02-15 14:11:09,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59550.73 MB 2025-02-15 14:11:09,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 14:11:09,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44547.67 MB 2025-02-15 14:11:09,570 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 14:11:09,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:11:09,577 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:11:09,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,578 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:11:09,578 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 14:11:09,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,579 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,579 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:11:09,585 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:11:09,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,586 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,586 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:11:09,586 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:11:09,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,587 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:11:09,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,587 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:11:09,588 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:11:09,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,588 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:11:09,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,592 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,593 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,594 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:11:09,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:11:09,606 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:28,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:28,355 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:28,360 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:12:28,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:28,361 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1727, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:12:28,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:28,362 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1727, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:12:54,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:12:54,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:12:54,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.60 seconds 2025-02-15 14:12:54,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:54,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38739.60 MB 2025-02-15 14:12:54,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44851.36 MB 2025-02-15 14:12:54,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.76 MB 2025-02-15 14:12:54,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72196.55 MB 2025-02-15 14:12:54,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55952.02 MB 2025-02-15 14:12:54,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16244.54 MB 2025-02-15 14:12:54,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53729.18 MB 2025-02-15 14:12:55,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:12:55,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:12:55,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 14:12:55,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:55,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44851.36 MB 2025-02-15 14:12:55,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38492.87 MB 2025-02-15 14:12:55,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6358.49 MB 2025-02-15 14:12:55,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55952.02 MB 2025-02-15 14:12:55,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67662.51 MB 2025-02-15 14:12:55,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11710.50 MB 2025-02-15 14:12:55,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62461.44 MB 2025-02-15 14:12:57,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:12:57,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:12:57,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 14:12:57,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38492.87 MB 2025-02-15 14:12:57,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39023.71 MB 2025-02-15 14:12:57,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:12:57,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67662.51 MB 2025-02-15 14:12:57,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55952.02 MB 2025-02-15 14:12:57,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11710.50 MB 2025-02-15 14:12:57,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43002.26 MB 2025-02-15 14:12:57,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:12:57,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:12:57,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:12:57,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39023.71 MB 2025-02-15 14:12:57,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40913.20 MB 2025-02-15 14:12:57,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:12:57,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55952.02 MB 2025-02-15 14:12:57,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55952.02 MB 2025-02-15 14:12:57,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:12:57,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42330.63 MB 2025-02-15 14:12:57,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:12:57,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:12:57,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:12:57,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40913.20 MB 2025-02-15 14:12:57,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43155.06 MB 2025-02-15 14:12:57,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:12:57,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55952.02 MB 2025-02-15 14:12:57,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55954.11 MB 2025-02-15 14:12:57,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:12:57,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48699.34 MB 2025-02-15 14:12:57,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:12:57,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:12:57,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:12:57,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39023.71 MB 2025-02-15 14:12:57,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43155.06 MB 2025-02-15 14:12:57,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:12:57,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55952.02 MB 2025-02-15 14:12:57,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55954.11 MB 2025-02-15 14:12:57,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:12:57,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48699.34 MB 2025-02-15 14:12:57,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:12:57,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:12:57,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:12:57,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43862.85 MB 2025-02-15 14:12:57,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44629.85 MB 2025-02-15 14:12:57,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:12:57,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55954.11 MB 2025-02-15 14:12:57,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56371.45 MB 2025-02-15 14:12:57,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:12:57,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45337.64 MB 2025-02-15 14:12:57,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:12:57,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:12:57,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:12:57,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45043.72 MB 2025-02-15 14:12:57,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45250.33 MB 2025-02-15 14:12:57,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.61 MB 2025-02-15 14:12:57,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56371.45 MB 2025-02-15 14:12:57,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56371.45 MB 2025-02-15 14:12:57,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:12:57,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45447.34 MB 2025-02-15 14:12:57,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:12:57,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:12:57,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.09 seconds 2025-02-15 14:12:57,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32722.59 MB 2025-02-15 14:12:57,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45450.91 MB 2025-02-15 14:12:57,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12728.32 MB 2025-02-15 14:12:57,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72196.55 MB 2025-02-15 14:12:57,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56371.45 MB 2025-02-15 14:12:57,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15825.11 MB 2025-02-15 14:12:57,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45450.91 MB 2025-02-15 14:12:57,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:12:57,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:12:57,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:12:57,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45450.91 MB 2025-02-15 14:12:57,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45551.13 MB 2025-02-15 14:12:57,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-15 14:12:57,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56371.45 MB 2025-02-15 14:12:57,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56371.45 MB 2025-02-15 14:12:57,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:12:57,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46152.46 MB 2025-02-15 14:12:57,744 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 14:12:57,745 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:12:57,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:12:57,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:12:57,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:12:57,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:12:57,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33985.02 MB 2025-02-15 14:12:57,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38169.25 MB 2025-02-15 14:12:57,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-15 14:12:57,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56371.45 MB 2025-02-15 14:12:57,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56371.45 MB 2025-02-15 14:12:57,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:12:57,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42353.07 MB 2025-02-15 14:12:57,913 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 14:12:57,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,915 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:12:57,920 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:12:57,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:12:57,922 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:12:57,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,922 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,923 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:57,929 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:12:57,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,929 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,930 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:57,930 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:12:57,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,930 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:57,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,931 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:12:57,931 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:12:57,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,931 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:12:57,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,935 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,936 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,937 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:12:57,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:12:57,948 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:13:47,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:13:47,689 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:13:47,696 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:13:47,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:13:47,698 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2076, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:13:47,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:13:47,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2076, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:14:19,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:14:19,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:14:19,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.26 seconds 2025-02-15 14:14:19,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:19,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41293.26 MB 2025-02-15 14:14:19,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48640.11 MB 2025-02-15 14:14:19,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7346.85 MB 2025-02-15 14:14:19,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69138.91 MB 2025-02-15 14:14:19,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63422.07 MB 2025-02-15 14:14:19,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5716.84 MB 2025-02-15 14:14:19,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57559.41 MB 2025-02-15 14:14:20,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:14:20,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:14:20,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 14:14:20,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:20,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48640.11 MB 2025-02-15 14:14:20,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40428.98 MB 2025-02-15 14:14:20,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8211.13 MB 2025-02-15 14:14:20,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63422.07 MB 2025-02-15 14:14:20,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77894.52 MB 2025-02-15 14:14:20,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14472.45 MB 2025-02-15 14:14:20,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70397.83 MB 2025-02-15 14:14:22,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:14:22,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:14:22,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:14:22,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40428.98 MB 2025-02-15 14:14:22,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40959.82 MB 2025-02-15 14:14:22,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:14:22,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77894.52 MB 2025-02-15 14:14:22,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51896.12 MB 2025-02-15 14:14:22,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25998.39 MB 2025-02-15 14:14:22,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44938.37 MB 2025-02-15 14:14:22,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:14:22,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:14:22,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:14:22,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40959.82 MB 2025-02-15 14:14:22,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42849.32 MB 2025-02-15 14:14:22,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:14:22,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51896.12 MB 2025-02-15 14:14:22,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51898.22 MB 2025-02-15 14:14:22,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:14:22,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44266.75 MB 2025-02-15 14:14:22,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:14:22,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:14:22,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:14:22,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42849.32 MB 2025-02-15 14:14:22,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45091.17 MB 2025-02-15 14:14:22,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:14:22,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51898.22 MB 2025-02-15 14:14:22,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52841.94 MB 2025-02-15 14:14:22,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 14:14:22,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50635.46 MB 2025-02-15 14:14:22,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:14:22,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:14:22,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:14:22,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40959.82 MB 2025-02-15 14:14:22,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45091.17 MB 2025-02-15 14:14:22,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:14:22,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51896.12 MB 2025-02-15 14:14:22,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52841.94 MB 2025-02-15 14:14:22,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 945.82 MB 2025-02-15 14:14:22,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50635.46 MB 2025-02-15 14:14:22,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:14:22,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:14:22,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:14:22,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45798.96 MB 2025-02-15 14:14:22,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46565.96 MB 2025-02-15 14:14:22,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:14:22,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52841.94 MB 2025-02-15 14:14:22,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53259.27 MB 2025-02-15 14:14:22,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:14:22,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47273.75 MB 2025-02-15 14:14:22,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:14:22,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:14:22,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:14:22,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46978.85 MB 2025-02-15 14:14:22,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47184.92 MB 2025-02-15 14:14:22,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.07 MB 2025-02-15 14:14:22,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53259.27 MB 2025-02-15 14:14:22,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53259.27 MB 2025-02-15 14:14:22,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:14:22,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47413.13 MB 2025-02-15 14:14:22,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:14:22,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:14:22,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.72 seconds 2025-02-15 14:14:22,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34060.31 MB 2025-02-15 14:14:22,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47385.80 MB 2025-02-15 14:14:22,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13325.49 MB 2025-02-15 14:14:22,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69138.91 MB 2025-02-15 14:14:22,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53259.27 MB 2025-02-15 14:14:22,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15879.63 MB 2025-02-15 14:14:22,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47413.13 MB 2025-02-15 14:14:22,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:14:22,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:14:22,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:14:22,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47385.80 MB 2025-02-15 14:14:22,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47486.17 MB 2025-02-15 14:14:22,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 14:14:22,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53259.27 MB 2025-02-15 14:14:22,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53259.27 MB 2025-02-15 14:14:22,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:14:22,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48088.44 MB 2025-02-15 14:14:22,705 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 14:14:22,705 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 14:14:22,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:14:22,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:14:22,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:14:22,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:14:22,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35323.04 MB 2025-02-15 14:14:22,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39513.42 MB 2025-02-15 14:14:22,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 14:14:22,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53259.27 MB 2025-02-15 14:14:22,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57449.38 MB 2025-02-15 14:14:22,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 14:14:22,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43703.53 MB 2025-02-15 14:14:22,875 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 14:14:22,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,877 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:14:22,882 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:14:22,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,883 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:14:22,883 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 14:14:22,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,884 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,885 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:14:22,891 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:14:22,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,891 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,892 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:14:22,892 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:14:22,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,892 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:14:22,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,893 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:14:22,893 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:14:22,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,893 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:14:22,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,899 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,901 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,903 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:14:22,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:14:22,957 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:15:51,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:15:51,492 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:15:51,497 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:15:51,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:15:51,498 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:15:51,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:15:51,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:16:12,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:16:12,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:16:12,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.59 seconds 2025-02-15 14:16:12,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:12,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36321.31 MB 2025-02-15 14:16:12,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41081.19 MB 2025-02-15 14:16:12,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4759.88 MB 2025-02-15 14:16:12,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70338.48 MB 2025-02-15 14:16:12,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56207.87 MB 2025-02-15 14:16:12,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14130.61 MB 2025-02-15 14:16:12,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50096.04 MB 2025-02-15 14:16:12,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:16:12,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:16:12,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 14:16:12,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:12,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41081.19 MB 2025-02-15 14:16:12,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36750.52 MB 2025-02-15 14:16:12,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4330.68 MB 2025-02-15 14:16:12,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56207.87 MB 2025-02-15 14:16:12,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60433.63 MB 2025-02-15 14:16:12,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4225.76 MB 2025-02-15 14:16:12,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53692.49 MB 2025-02-15 14:16:14,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:16:14,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:16:14,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:16:14,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36750.52 MB 2025-02-15 14:16:14,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37281.36 MB 2025-02-15 14:16:14,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:16:14,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60433.63 MB 2025-02-15 14:16:14,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52030.34 MB 2025-02-15 14:16:14,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8403.29 MB 2025-02-15 14:16:14,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41259.91 MB 2025-02-15 14:16:14,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:16:14,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:16:14,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:16:14,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37281.36 MB 2025-02-15 14:16:14,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39170.75 MB 2025-02-15 14:16:14,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.39 MB 2025-02-15 14:16:14,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 14:16:14,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52030.34 MB 2025-02-15 14:16:14,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:16:14,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40588.18 MB 2025-02-15 14:16:14,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:16:14,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:16:14,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:16:14,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39170.75 MB 2025-02-15 14:16:14,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41412.60 MB 2025-02-15 14:16:14,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:16:14,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 14:16:14,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52030.34 MB 2025-02-15 14:16:14,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:16:14,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46956.89 MB 2025-02-15 14:16:14,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:16:14,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:16:14,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:16:14,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37281.36 MB 2025-02-15 14:16:14,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41412.60 MB 2025-02-15 14:16:14,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.25 MB 2025-02-15 14:16:14,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 14:16:14,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52030.34 MB 2025-02-15 14:16:14,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:16:14,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46956.89 MB 2025-02-15 14:16:14,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:16:14,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:16:14,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:16:14,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42120.39 MB 2025-02-15 14:16:14,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42887.39 MB 2025-02-15 14:16:14,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:16:14,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52030.34 MB 2025-02-15 14:16:14,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52447.67 MB 2025-02-15 14:16:14,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:16:14,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43595.18 MB 2025-02-15 14:16:14,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:16:14,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:16:14,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:16:14,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43300.28 MB 2025-02-15 14:16:14,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43507.36 MB 2025-02-15 14:16:14,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.07 MB 2025-02-15 14:16:14,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52447.67 MB 2025-02-15 14:16:14,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52447.67 MB 2025-02-15 14:16:14,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:16:14,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43731.94 MB 2025-02-15 14:16:14,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:16:14,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:16:14,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.00 seconds 2025-02-15 14:16:14,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31635.22 MB 2025-02-15 14:16:14,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43708.43 MB 2025-02-15 14:16:14,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12073.20 MB 2025-02-15 14:16:14,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70338.48 MB 2025-02-15 14:16:14,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52447.67 MB 2025-02-15 14:16:14,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17890.80 MB 2025-02-15 14:16:14,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43731.94 MB 2025-02-15 14:16:14,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:16:14,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:16:14,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:16:14,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43708.43 MB 2025-02-15 14:16:14,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43808.90 MB 2025-02-15 14:16:14,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:16:14,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52447.67 MB 2025-02-15 14:16:14,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52447.67 MB 2025-02-15 14:16:14,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:16:14,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44411.70 MB 2025-02-15 14:16:14,786 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:16:14,786 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:16:14,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:16:14,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:16:14,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:16:14,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:16:14,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32898.14 MB 2025-02-15 14:16:14,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37092.63 MB 2025-02-15 14:16:14,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:16:14,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52447.67 MB 2025-02-15 14:16:14,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56641.98 MB 2025-02-15 14:16:14,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 14:16:14,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41286.93 MB 2025-02-15 14:16:14,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:16:14,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:16:14,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:16:14,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,960 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:16:14,960 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:16:14,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,961 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,962 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:16:14,967 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:16:14,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,968 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,969 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:16:14,969 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:16:14,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,969 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:16:14,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,970 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:16:14,970 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:16:14,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,970 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:16:14,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,976 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,977 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,978 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:16:14,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:16:14,990 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:10,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:10,968 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:10,973 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:17:10,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:10,974 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1982, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:17:10,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:10,975 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1982, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:17:41,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:17:41,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:17:41,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.65 seconds 2025-02-15 14:17:41,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:41,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40881.80 MB 2025-02-15 14:17:41,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47895.99 MB 2025-02-15 14:17:41,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7014.19 MB 2025-02-15 14:17:41,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69652.71 MB 2025-02-15 14:17:41,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56346.28 MB 2025-02-15 14:17:41,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13306.43 MB 2025-02-15 14:17:41,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56694.96 MB 2025-02-15 14:17:41,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:17:41,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:17:41,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 14:17:41,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:41,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47895.99 MB 2025-02-15 14:17:41,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40183.86 MB 2025-02-15 14:17:41,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7712.14 MB 2025-02-15 14:17:41,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56346.28 MB 2025-02-15 14:17:41,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69264.74 MB 2025-02-15 14:17:41,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12918.46 MB 2025-02-15 14:17:41,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67324.49 MB 2025-02-15 14:17:43,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:17:43,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:17:43,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:17:43,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:43,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40183.86 MB 2025-02-15 14:17:43,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40714.70 MB 2025-02-15 14:17:43,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:17:43,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69264.74 MB 2025-02-15 14:17:43,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52162.46 MB 2025-02-15 14:17:43,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17102.27 MB 2025-02-15 14:17:43,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44693.24 MB 2025-02-15 14:17:43,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:17:43,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:17:43,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:17:43,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:43,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40714.70 MB 2025-02-15 14:17:43,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42604.02 MB 2025-02-15 14:17:43,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.33 MB 2025-02-15 14:17:43,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52162.46 MB 2025-02-15 14:17:43,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52162.46 MB 2025-02-15 14:17:43,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:17:43,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44021.45 MB 2025-02-15 14:17:43,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:17:43,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:17:43,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:17:43,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:43,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42604.02 MB 2025-02-15 14:17:43,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44845.88 MB 2025-02-15 14:17:43,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:17:43,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52162.46 MB 2025-02-15 14:17:43,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52162.46 MB 2025-02-15 14:17:43,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:17:43,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50390.16 MB 2025-02-15 14:17:43,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:17:43,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:17:43,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:17:43,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:43,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40714.70 MB 2025-02-15 14:17:43,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44845.88 MB 2025-02-15 14:17:43,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.18 MB 2025-02-15 14:17:43,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52162.46 MB 2025-02-15 14:17:43,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52162.46 MB 2025-02-15 14:17:43,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:17:43,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50390.16 MB 2025-02-15 14:17:44,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:17:44,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:17:44,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:17:44,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:44,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45553.67 MB 2025-02-15 14:17:44,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46320.67 MB 2025-02-15 14:17:44,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:17:44,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52162.46 MB 2025-02-15 14:17:44,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52581.89 MB 2025-02-15 14:17:44,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 14:17:44,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47028.46 MB 2025-02-15 14:17:44,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:17:44,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:17:44,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:17:44,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:44,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46733.56 MB 2025-02-15 14:17:44,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46940.91 MB 2025-02-15 14:17:44,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.35 MB 2025-02-15 14:17:44,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52581.89 MB 2025-02-15 14:17:44,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52581.89 MB 2025-02-15 14:17:44,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:17:44,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47139.02 MB 2025-02-15 14:17:44,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:17:44,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:17:44,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.14 seconds 2025-02-15 14:17:44,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:44,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33976.36 MB 2025-02-15 14:17:44,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47141.98 MB 2025-02-15 14:17:44,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13165.63 MB 2025-02-15 14:17:44,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69652.71 MB 2025-02-15 14:17:44,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52581.89 MB 2025-02-15 14:17:44,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17070.82 MB 2025-02-15 14:17:44,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47141.98 MB 2025-02-15 14:17:44,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:17:44,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:17:44,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:17:44,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:44,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47141.98 MB 2025-02-15 14:17:44,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47242.45 MB 2025-02-15 14:17:44,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:17:44,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52581.89 MB 2025-02-15 14:17:44,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52581.89 MB 2025-02-15 14:17:44,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:17:44,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47845.25 MB 2025-02-15 14:17:44,407 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:17:44,408 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:17:44,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:17:44,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:17:44,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:17:44,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:17:44,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35239.28 MB 2025-02-15 14:17:44,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39433.76 MB 2025-02-15 14:17:44,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:17:44,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52581.89 MB 2025-02-15 14:17:44,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56778.29 MB 2025-02-15 14:17:44,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 14:17:44,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43628.07 MB 2025-02-15 14:17:44,577 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:17:44,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,578 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,579 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:17:44,584 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:17:44,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,585 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:17:44,585 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:17:44,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,586 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,586 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:44,592 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:17:44,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,593 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,593 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:44,593 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:17:44,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,594 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:44,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,594 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:17:44,594 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:17:44,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,595 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:17:44,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,601 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,603 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,605 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:17:44,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:17:44,618 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:22,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:22,766 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:22,771 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:19:22,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:22,772 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:19:22,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:22,773 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:19:43,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:19:43,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:19:43,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.74 seconds 2025-02-15 14:19:43,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:43,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36564.86 MB 2025-02-15 14:19:43,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41324.74 MB 2025-02-15 14:19:43,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4759.88 MB 2025-02-15 14:19:43,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69910.66 MB 2025-02-15 14:19:43,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48091.89 MB 2025-02-15 14:19:43,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21818.77 MB 2025-02-15 14:19:43,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50339.59 MB 2025-02-15 14:19:43,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:19:43,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:19:43,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 14:19:43,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:43,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41324.74 MB 2025-02-15 14:19:43,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36994.06 MB 2025-02-15 14:19:43,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4330.68 MB 2025-02-15 14:19:43,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48091.89 MB 2025-02-15 14:19:43,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55647.93 MB 2025-02-15 14:19:43,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7556.04 MB 2025-02-15 14:19:43,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51688.81 MB 2025-02-15 14:19:45,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:19:45,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:19:45,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 14:19:45,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36994.06 MB 2025-02-15 14:19:45,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37524.91 MB 2025-02-15 14:19:45,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:19:45,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55647.93 MB 2025-02-15 14:19:45,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49507.47 MB 2025-02-15 14:19:45,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6140.46 MB 2025-02-15 14:19:45,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41503.45 MB 2025-02-15 14:19:45,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:19:45,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:19:45,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:19:45,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37524.91 MB 2025-02-15 14:19:45,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39414.40 MB 2025-02-15 14:19:45,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:19:45,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-15 14:19:45,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49507.47 MB 2025-02-15 14:19:45,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:19:45,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40831.83 MB 2025-02-15 14:19:45,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:19:45,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:19:45,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:19:45,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39414.40 MB 2025-02-15 14:19:45,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41656.25 MB 2025-02-15 14:19:45,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:19:45,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-15 14:19:45,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49507.47 MB 2025-02-15 14:19:45,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:19:45,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47200.54 MB 2025-02-15 14:19:45,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:19:45,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:19:45,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:19:45,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37524.91 MB 2025-02-15 14:19:45,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41656.25 MB 2025-02-15 14:19:45,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:19:45,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-15 14:19:45,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49507.47 MB 2025-02-15 14:19:45,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:19:45,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47200.54 MB 2025-02-15 14:19:45,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:19:45,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:19:45,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:19:45,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42364.04 MB 2025-02-15 14:19:45,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43131.05 MB 2025-02-15 14:19:45,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:19:45,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-15 14:19:45,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49922.70 MB 2025-02-15 14:19:45,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 14:19:45,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43838.83 MB 2025-02-15 14:19:45,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:19:45,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:19:45,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:19:45,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43543.93 MB 2025-02-15 14:19:45,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43751.86 MB 2025-02-15 14:19:45,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.92 MB 2025-02-15 14:19:45,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49922.70 MB 2025-02-15 14:19:45,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49922.70 MB 2025-02-15 14:19:45,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:19:45,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43970.93 MB 2025-02-15 14:19:45,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:19:45,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:19:45,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.15 seconds 2025-02-15 14:19:45,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:45,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31878.77 MB 2025-02-15 14:19:45,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43952.93 MB 2025-02-15 14:19:45,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12074.16 MB 2025-02-15 14:19:45,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69910.66 MB 2025-02-15 14:19:45,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49922.70 MB 2025-02-15 14:19:45,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19987.96 MB 2025-02-15 14:19:45,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43970.93 MB 2025-02-15 14:19:46,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:19:46,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:19:46,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:19:46,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:46,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43952.93 MB 2025-02-15 14:19:46,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44053.40 MB 2025-02-15 14:19:46,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:19:46,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49922.70 MB 2025-02-15 14:19:46,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49922.70 MB 2025-02-15 14:19:46,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:19:46,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44656.20 MB 2025-02-15 14:19:46,207 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:19:46,207 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:19:46,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:19:46,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:19:46,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 14:19:46,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:19:46,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33141.69 MB 2025-02-15 14:19:46,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37336.18 MB 2025-02-15 14:19:46,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:19:46,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49922.70 MB 2025-02-15 14:19:46,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54117.01 MB 2025-02-15 14:19:46,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-15 14:19:46,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41530.48 MB 2025-02-15 14:19:46,408 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:19:46,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,410 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:19:46,415 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:19:46,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,416 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:19:46,417 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:19:46,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,418 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,418 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:46,424 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:19:46,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,424 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,425 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:46,425 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:19:46,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,425 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:46,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,426 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:19:46,426 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:19:46,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,426 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:19:46,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,431 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,432 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,433 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:19:46,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:19:46,445 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:19,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:19,601 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:19,606 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:21:19,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:19,608 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2017, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:21:19,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:19,609 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2017, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:21:50,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:21:50,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:21:50,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.16 seconds 2025-02-15 14:21:50,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:50,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41368.19 MB 2025-02-15 14:21:50,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48506.89 MB 2025-02-15 14:21:50,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7138.71 MB 2025-02-15 14:21:50,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67371.01 MB 2025-02-15 14:21:50,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60274.25 MB 2025-02-15 14:21:50,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7096.76 MB 2025-02-15 14:21:50,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57407.84 MB 2025-02-15 14:21:50,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:21:50,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:21:50,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 14:21:50,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:50,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48506.89 MB 2025-02-15 14:21:50,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40608.31 MB 2025-02-15 14:21:50,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7898.59 MB 2025-02-15 14:21:50,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60274.25 MB 2025-02-15 14:21:50,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73847.01 MB 2025-02-15 14:21:50,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13572.77 MB 2025-02-15 14:21:50,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68916.19 MB 2025-02-15 14:21:52,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:21:52,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:21:52,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:21:52,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:52,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40608.31 MB 2025-02-15 14:21:52,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41139.15 MB 2025-02-15 14:21:52,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:21:52,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73847.01 MB 2025-02-15 14:21:52,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48945.43 MB 2025-02-15 14:21:52,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24901.58 MB 2025-02-15 14:21:52,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45117.69 MB 2025-02-15 14:21:52,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:21:52,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:21:52,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:21:52,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:52,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41139.15 MB 2025-02-15 14:21:52,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43028.34 MB 2025-02-15 14:21:52,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.19 MB 2025-02-15 14:21:52,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48945.43 MB 2025-02-15 14:21:52,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48945.43 MB 2025-02-15 14:21:52,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:21:52,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44445.77 MB 2025-02-15 14:21:53,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:21:53,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:21:53,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:21:53,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43028.34 MB 2025-02-15 14:21:53,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45270.20 MB 2025-02-15 14:21:53,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:21:53,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48945.43 MB 2025-02-15 14:21:53,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52720.30 MB 2025-02-15 14:21:53,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 14:21:53,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50814.48 MB 2025-02-15 14:21:53,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:21:53,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:21:53,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:21:53,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41139.15 MB 2025-02-15 14:21:53,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45270.20 MB 2025-02-15 14:21:53,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.05 MB 2025-02-15 14:21:53,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48945.43 MB 2025-02-15 14:21:53,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52720.30 MB 2025-02-15 14:21:53,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 14:21:53,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50814.48 MB 2025-02-15 14:21:53,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:21:53,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:21:53,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:21:53,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45977.99 MB 2025-02-15 14:21:53,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46744.99 MB 2025-02-15 14:21:53,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:21:53,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52720.30 MB 2025-02-15 14:21:53,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53139.73 MB 2025-02-15 14:21:53,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 14:21:53,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47452.78 MB 2025-02-15 14:21:53,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:21:53,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:21:53,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:21:53,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47157.88 MB 2025-02-15 14:21:53,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47364.92 MB 2025-02-15 14:21:53,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.05 MB 2025-02-15 14:21:53,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53139.73 MB 2025-02-15 14:21:53,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53139.73 MB 2025-02-15 14:21:53,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:21:53,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47585.43 MB 2025-02-15 14:21:53,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:21:53,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:21:53,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.64 seconds 2025-02-15 14:21:53,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34340.80 MB 2025-02-15 14:21:53,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47565.87 MB 2025-02-15 14:21:53,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13225.08 MB 2025-02-15 14:21:53,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67371.01 MB 2025-02-15 14:21:53,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53139.73 MB 2025-02-15 14:21:53,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14231.27 MB 2025-02-15 14:21:53,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47585.43 MB 2025-02-15 14:21:53,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:21:53,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:21:53,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:21:53,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47565.87 MB 2025-02-15 14:21:53,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47666.28 MB 2025-02-15 14:21:53,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.41 MB 2025-02-15 14:21:53,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53139.73 MB 2025-02-15 14:21:53,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53139.73 MB 2025-02-15 14:21:53,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:21:53,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48268.71 MB 2025-02-15 14:21:53,534 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 14:21:53,535 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:21:53,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:21:53,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:21:53,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:21:53,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:21:53,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35603.59 MB 2025-02-15 14:21:53,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39795.51 MB 2025-02-15 14:21:53,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4191.92 MB 2025-02-15 14:21:53,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53139.73 MB 2025-02-15 14:21:53,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57331.94 MB 2025-02-15 14:21:53,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 14:21:53,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43987.72 MB 2025-02-15 14:21:53,700 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 14:21:53,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,701 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,702 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:21:53,707 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:21:53,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,708 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:21:53,708 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:21:53,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,709 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,709 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:53,715 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:21:53,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,716 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,716 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:53,716 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:21:53,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,717 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:53,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,717 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:21:53,717 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:21:53,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,718 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:21:53,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,722 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,724 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,725 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:21:53,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:21:53,740 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:21,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:21,178 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:21,183 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:23:21,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:21,184 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2113, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:23:21,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:21,185 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2113, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:23:53,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:23:53,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:23:53,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.78 seconds 2025-02-15 14:23:53,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:53,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42158.86 MB 2025-02-15 14:23:53,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49637.30 MB 2025-02-15 14:23:53,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7478.44 MB 2025-02-15 14:23:53,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70707.58 MB 2025-02-15 14:23:53,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60859.35 MB 2025-02-15 14:23:53,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9848.23 MB 2025-02-15 14:23:53,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58651.82 MB 2025-02-15 14:23:54,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:23:54,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:23:54,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 14:23:54,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:54,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49637.30 MB 2025-02-15 14:23:54,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41229.11 MB 2025-02-15 14:23:54,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8408.19 MB 2025-02-15 14:23:54,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60859.35 MB 2025-02-15 14:23:54,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75176.61 MB 2025-02-15 14:23:54,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14317.26 MB 2025-02-15 14:23:54,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 71165.76 MB 2025-02-15 14:23:56,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:23:56,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:23:56,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-15 14:23:56,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41229.11 MB 2025-02-15 14:23:56,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41759.95 MB 2025-02-15 14:23:56,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:23:56,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75176.61 MB 2025-02-15 14:23:56,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49186.60 MB 2025-02-15 14:23:56,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25990.00 MB 2025-02-15 14:23:56,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45738.49 MB 2025-02-15 14:23:56,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:23:56,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:23:56,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:23:56,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41759.95 MB 2025-02-15 14:23:56,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43649.08 MB 2025-02-15 14:23:56,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.13 MB 2025-02-15 14:23:56,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49186.60 MB 2025-02-15 14:23:56,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49186.60 MB 2025-02-15 14:23:56,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:23:56,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45066.50 MB 2025-02-15 14:23:56,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:23:56,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:23:56,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:23:56,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43649.08 MB 2025-02-15 14:23:56,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45890.93 MB 2025-02-15 14:23:56,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:23:56,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49186.60 MB 2025-02-15 14:23:56,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53905.20 MB 2025-02-15 14:23:56,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 14:23:56,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51435.21 MB 2025-02-15 14:23:56,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:23:56,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:23:56,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:23:56,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41759.95 MB 2025-02-15 14:23:56,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45890.93 MB 2025-02-15 14:23:56,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.98 MB 2025-02-15 14:23:56,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49186.60 MB 2025-02-15 14:23:56,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53905.20 MB 2025-02-15 14:23:56,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 14:23:56,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51435.21 MB 2025-02-15 14:23:56,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:23:56,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:23:56,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:23:56,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46598.72 MB 2025-02-15 14:23:56,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47365.72 MB 2025-02-15 14:23:56,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:23:56,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53905.20 MB 2025-02-15 14:23:56,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-15 14:23:56,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 14:23:56,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48073.51 MB 2025-02-15 14:23:56,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:23:56,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:23:56,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:23:56,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47778.61 MB 2025-02-15 14:23:56,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47985.42 MB 2025-02-15 14:23:56,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.81 MB 2025-02-15 14:23:56,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54324.63 MB 2025-02-15 14:23:56,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-15 14:23:56,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:23:56,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48197.72 MB 2025-02-15 14:23:56,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:23:56,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:23:56,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.34 seconds 2025-02-15 14:23:56,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34796.99 MB 2025-02-15 14:23:56,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48186.20 MB 2025-02-15 14:23:56,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13389.21 MB 2025-02-15 14:23:56,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70707.58 MB 2025-02-15 14:23:56,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-15 14:23:56,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16382.95 MB 2025-02-15 14:23:56,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48197.72 MB 2025-02-15 14:23:56,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:23:56,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:23:56,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:23:56,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48186.20 MB 2025-02-15 14:23:56,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48286.52 MB 2025-02-15 14:23:56,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-15 14:23:56,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54324.63 MB 2025-02-15 14:23:56,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-15 14:23:56,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:23:56,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48888.44 MB 2025-02-15 14:23:56,807 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 14:23:56,808 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:23:56,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:23:56,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:23:56,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:23:56,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:23:56,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36059.62 MB 2025-02-15 14:23:56,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40247.95 MB 2025-02-15 14:23:56,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-15 14:23:56,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54324.63 MB 2025-02-15 14:23:56,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-15 14:23:56,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:23:56,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44435.76 MB 2025-02-15 14:23:56,972 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 14:23:56,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,974 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:56,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,975 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:23:56,979 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:23:56,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,980 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:23:56,980 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:23:56,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,981 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:56,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,982 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:56,987 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:23:56,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,988 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:56,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,989 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:56,989 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:23:56,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,989 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:56,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:23:56,990 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:23:56,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:23:56,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,993 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:56,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,994 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:56,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:56,995 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:23:57,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:23:57,007 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:08,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:08,792 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:08,797 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:25:08,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:08,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1894, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:25:08,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:08,799 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1894, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:25:37,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:25:37,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:25:37,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.17 seconds 2025-02-15 14:25:37,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:37,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40754.55 MB 2025-02-15 14:25:37,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47457.31 MB 2025-02-15 14:25:37,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6702.76 MB 2025-02-15 14:25:37,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67821.90 MB 2025-02-15 14:25:37,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62683.87 MB 2025-02-15 14:25:37,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5138.02 MB 2025-02-15 14:25:37,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56341.22 MB 2025-02-15 14:25:38,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:25:38,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:25:38,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 14:25:38,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:38,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47457.31 MB 2025-02-15 14:25:38,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40212.32 MB 2025-02-15 14:25:38,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7245.00 MB 2025-02-15 14:25:38,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62683.87 MB 2025-02-15 14:25:38,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75084.33 MB 2025-02-15 14:25:38,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12400.46 MB 2025-02-15 14:25:38,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66099.40 MB 2025-02-15 14:25:40,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:25:40,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:25:40,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:25:40,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40212.32 MB 2025-02-15 14:25:40,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40743.16 MB 2025-02-15 14:25:40,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:25:40,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75084.33 MB 2025-02-15 14:25:40,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58491.67 MB 2025-02-15 14:25:40,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16592.67 MB 2025-02-15 14:25:40,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44721.71 MB 2025-02-15 14:25:40,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:25:40,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:25:40,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:25:40,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40743.16 MB 2025-02-15 14:25:40,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42632.65 MB 2025-02-15 14:25:40,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:25:40,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58491.67 MB 2025-02-15 14:25:40,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58491.67 MB 2025-02-15 14:25:40,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44050.08 MB 2025-02-15 14:25:40,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:25:40,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:25:40,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-15 14:25:40,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42632.65 MB 2025-02-15 14:25:40,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44874.51 MB 2025-02-15 14:25:40,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:25:40,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58491.67 MB 2025-02-15 14:25:40,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58491.67 MB 2025-02-15 14:25:40,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50418.79 MB 2025-02-15 14:25:40,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:25:40,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:25:40,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.36 seconds 2025-02-15 14:25:40,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40743.16 MB 2025-02-15 14:25:40,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44874.51 MB 2025-02-15 14:25:40,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:25:40,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58491.67 MB 2025-02-15 14:25:40,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58491.67 MB 2025-02-15 14:25:40,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50418.79 MB 2025-02-15 14:25:40,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:25:40,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:25:40,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:25:40,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45582.30 MB 2025-02-15 14:25:40,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46349.30 MB 2025-02-15 14:25:40,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:25:40,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58491.67 MB 2025-02-15 14:25:40,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58911.10 MB 2025-02-15 14:25:40,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 14:25:40,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47057.09 MB 2025-02-15 14:25:40,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:25:40,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:25:40,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:25:40,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46762.19 MB 2025-02-15 14:25:40,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46966.30 MB 2025-02-15 14:25:40,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.11 MB 2025-02-15 14:25:40,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58911.10 MB 2025-02-15 14:25:40,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58911.10 MB 2025-02-15 14:25:40,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47176.43 MB 2025-02-15 14:25:40,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:25:40,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:25:40,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.76 seconds 2025-02-15 14:25:40,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34155.71 MB 2025-02-15 14:25:40,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47167.13 MB 2025-02-15 14:25:40,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13011.42 MB 2025-02-15 14:25:40,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67821.90 MB 2025-02-15 14:25:40,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58911.10 MB 2025-02-15 14:25:40,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8910.80 MB 2025-02-15 14:25:40,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47176.43 MB 2025-02-15 14:25:40,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:25:40,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:25:40,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:25:40,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47167.13 MB 2025-02-15 14:25:40,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35418.41 MB 2025-02-15 14:25:40,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11748.72 MB 2025-02-15 14:25:40,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58911.10 MB 2025-02-15 14:25:40,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58911.10 MB 2025-02-15 14:25:40,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47401.27 MB 2025-02-15 14:25:40,850 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 14:25:40,850 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:25:40,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:25:40,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:25:40,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:25:40,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:25:40,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35418.41 MB 2025-02-15 14:25:40,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39607.76 MB 2025-02-15 14:25:40,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4189.36 MB 2025-02-15 14:25:40,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58911.10 MB 2025-02-15 14:25:40,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58911.10 MB 2025-02-15 14:25:40,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:25:40,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43796.60 MB 2025-02-15 14:25:41,015 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 14:25:41,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,016 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,017 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:25:41,021 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:25:41,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,022 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:25:41,023 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:25:41,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,023 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,024 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:41,030 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:25:41,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,030 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,031 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:41,031 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:25:41,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,031 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:41,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,032 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:25:41,032 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:25:41,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,032 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:25:41,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,037 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,038 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,039 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:25:41,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:25:41,058 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:19,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:19,645 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:19,649 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:27:19,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:19,651 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1663, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:27:19,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:19,652 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1663, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:27:45,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:27:45,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:27:45,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.78 seconds 2025-02-15 14:27:45,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:45,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39266.64 MB 2025-02-15 14:27:45,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45151.90 MB 2025-02-15 14:27:45,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5885.26 MB 2025-02-15 14:27:45,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72530.00 MB 2025-02-15 14:27:45,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51246.01 MB 2025-02-15 14:27:45,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21284.00 MB 2025-02-15 14:27:45,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54030.06 MB 2025-02-15 14:27:45,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:27:45,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:27:45,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 14:27:45,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:45,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45151.90 MB 2025-02-15 14:27:45,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39133.15 MB 2025-02-15 14:27:45,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6018.75 MB 2025-02-15 14:27:45,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51246.01 MB 2025-02-15 14:27:45,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63990.40 MB 2025-02-15 14:27:45,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12744.39 MB 2025-02-15 14:27:45,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62189.08 MB 2025-02-15 14:27:47,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:27:47,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:27:47,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:27:47,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39133.15 MB 2025-02-15 14:27:47,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39663.99 MB 2025-02-15 14:27:47,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:27:47,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63990.40 MB 2025-02-15 14:27:47,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46774.88 MB 2025-02-15 14:27:47,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17215.52 MB 2025-02-15 14:27:47,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43642.54 MB 2025-02-15 14:27:47,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:27:47,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:27:47,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:27:47,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39663.99 MB 2025-02-15 14:27:47,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41553.48 MB 2025-02-15 14:27:47,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:27:47,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46774.88 MB 2025-02-15 14:27:47,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46774.88 MB 2025-02-15 14:27:47,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:27:47,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42970.91 MB 2025-02-15 14:27:47,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:27:47,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:27:47,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 14:27:47,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41553.48 MB 2025-02-15 14:27:47,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43795.34 MB 2025-02-15 14:27:47,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:27:47,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46774.88 MB 2025-02-15 14:27:47,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51493.47 MB 2025-02-15 14:27:47,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 14:27:47,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49339.62 MB 2025-02-15 14:27:47,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:27:47,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:27:47,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:27:47,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39663.99 MB 2025-02-15 14:27:47,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43795.34 MB 2025-02-15 14:27:47,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:27:47,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46774.88 MB 2025-02-15 14:27:47,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51493.47 MB 2025-02-15 14:27:47,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 14:27:47,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49339.62 MB 2025-02-15 14:27:47,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:27:47,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:27:47,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:27:47,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44503.13 MB 2025-02-15 14:27:47,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45270.13 MB 2025-02-15 14:27:47,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:27:47,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51493.47 MB 2025-02-15 14:27:47,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51912.90 MB 2025-02-15 14:27:47,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-15 14:27:47,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45977.92 MB 2025-02-15 14:27:47,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:27:47,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:27:47,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:27:47,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45683.02 MB 2025-02-15 14:27:47,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45888.00 MB 2025-02-15 14:27:47,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.98 MB 2025-02-15 14:27:47,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51912.90 MB 2025-02-15 14:27:47,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51912.90 MB 2025-02-15 14:27:47,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:27:47,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46099.01 MB 2025-02-15 14:27:47,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:27:47,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:27:47,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.27 seconds 2025-02-15 14:27:47,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:47,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33472.61 MB 2025-02-15 14:27:47,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46087.94 MB 2025-02-15 14:27:47,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12615.33 MB 2025-02-15 14:27:47,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72530.00 MB 2025-02-15 14:27:47,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51912.90 MB 2025-02-15 14:27:47,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20617.10 MB 2025-02-15 14:27:47,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46099.01 MB 2025-02-15 14:27:48,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:27:48,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:27:48,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:27:48,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:48,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46087.94 MB 2025-02-15 14:27:48,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46187.84 MB 2025-02-15 14:27:48,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.90 MB 2025-02-15 14:27:48,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51912.90 MB 2025-02-15 14:27:48,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51912.90 MB 2025-02-15 14:27:48,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:27:48,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46787.25 MB 2025-02-15 14:27:48,206 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 14:27:48,206 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:27:48,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:27:48,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:27:48,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 14:27:48,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:27:48,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34734.40 MB 2025-02-15 14:27:48,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38905.29 MB 2025-02-15 14:27:48,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4170.89 MB 2025-02-15 14:27:48,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51912.90 MB 2025-02-15 14:27:48,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56084.14 MB 2025-02-15 14:27:48,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 14:27:48,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43076.52 MB 2025-02-15 14:27:48,433 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 14:27:48,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,434 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,435 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:27:48,440 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:27:48,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,441 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:27:48,441 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:27:48,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,442 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,442 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:48,448 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:27:48,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,449 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,449 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:48,449 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:27:48,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,450 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:48,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,450 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:27:48,450 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:27:48,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,451 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:27:48,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,455 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,456 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,457 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:27:48,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:27:48,469 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:21,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:21,839 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:21,845 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:29:21,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:21,846 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1996, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:29:21,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:21,847 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1996, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:29:52,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:29:52,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:29:52,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.90 seconds 2025-02-15 14:29:52,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:52,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41708.76 MB 2025-02-15 14:29:52,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48772.49 MB 2025-02-15 14:29:52,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7063.73 MB 2025-02-15 14:29:52,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69824.68 MB 2025-02-15 14:29:52,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56960.75 MB 2025-02-15 14:29:52,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12863.93 MB 2025-02-15 14:29:52,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57748.41 MB 2025-02-15 14:29:52,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:29:52,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:29:52,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 14:29:52,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:52,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48772.49 MB 2025-02-15 14:29:52,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40986.04 MB 2025-02-15 14:29:52,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7786.45 MB 2025-02-15 14:29:52,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56960.75 MB 2025-02-15 14:29:52,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70401.39 MB 2025-02-15 14:29:52,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13440.65 MB 2025-02-15 14:29:52,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68979.62 MB 2025-02-15 14:29:54,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:29:54,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:29:54,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:29:54,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:54,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40986.04 MB 2025-02-15 14:29:54,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41516.88 MB 2025-02-15 14:29:54,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:29:54,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70401.39 MB 2025-02-15 14:29:54,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49895.44 MB 2025-02-15 14:29:54,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20505.95 MB 2025-02-15 14:29:54,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45495.43 MB 2025-02-15 14:29:54,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:29:54,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:29:54,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:29:54,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:54,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41516.88 MB 2025-02-15 14:29:54,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43406.37 MB 2025-02-15 14:29:54,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:29:54,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49895.44 MB 2025-02-15 14:29:54,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49895.44 MB 2025-02-15 14:29:54,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:29:54,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44823.80 MB 2025-02-15 14:29:55,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:29:55,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:29:55,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:29:55,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43406.37 MB 2025-02-15 14:29:55,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45648.23 MB 2025-02-15 14:29:55,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:29:55,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49895.44 MB 2025-02-15 14:29:55,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53200.55 MB 2025-02-15 14:29:55,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 14:29:55,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51192.51 MB 2025-02-15 14:29:55,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:29:55,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:29:55,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:29:55,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41516.88 MB 2025-02-15 14:29:55,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45648.23 MB 2025-02-15 14:29:55,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:29:55,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49895.44 MB 2025-02-15 14:29:55,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53200.55 MB 2025-02-15 14:29:55,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 14:29:55,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51192.51 MB 2025-02-15 14:29:55,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:29:55,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:29:55,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 14:29:55,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46356.02 MB 2025-02-15 14:29:55,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47123.02 MB 2025-02-15 14:29:55,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:29:55,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53200.55 MB 2025-02-15 14:29:55,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53617.89 MB 2025-02-15 14:29:55,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:29:55,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47830.81 MB 2025-02-15 14:29:55,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:29:55,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:29:55,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:29:55,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47535.91 MB 2025-02-15 14:29:55,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47741.99 MB 2025-02-15 14:29:55,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.08 MB 2025-02-15 14:29:55,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53617.89 MB 2025-02-15 14:29:55,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53617.89 MB 2025-02-15 14:29:55,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:29:55,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47959.67 MB 2025-02-15 14:29:55,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:29:55,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:29:55,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.58 seconds 2025-02-15 14:29:55,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34754.53 MB 2025-02-15 14:29:55,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47942.52 MB 2025-02-15 14:29:55,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13187.98 MB 2025-02-15 14:29:55,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69824.68 MB 2025-02-15 14:29:55,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53617.89 MB 2025-02-15 14:29:55,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16206.79 MB 2025-02-15 14:29:55,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47959.67 MB 2025-02-15 14:29:55,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:29:55,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:29:55,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:29:55,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47942.52 MB 2025-02-15 14:29:55,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48042.72 MB 2025-02-15 14:29:55,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.20 MB 2025-02-15 14:29:55,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53617.89 MB 2025-02-15 14:29:55,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53617.89 MB 2025-02-15 14:29:55,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:29:55,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48644.36 MB 2025-02-15 14:29:55,713 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 14:29:55,714 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 14:29:55,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:29:55,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:29:55,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:29:55,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:29:55,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36016.91 MB 2025-02-15 14:29:55,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40200.11 MB 2025-02-15 14:29:55,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4183.20 MB 2025-02-15 14:29:55,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53617.89 MB 2025-02-15 14:29:55,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53617.89 MB 2025-02-15 14:29:55,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:29:55,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44382.80 MB 2025-02-15 14:29:55,881 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 14:29:55,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,882 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,883 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:29:55,888 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:29:55,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,889 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:29:55,889 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 14:29:55,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,890 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,890 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:55,896 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:29:55,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,897 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,897 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:55,897 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:29:55,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,898 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:55,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,898 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:29:55,898 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:29:55,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,899 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:29:55,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,902 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,903 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,904 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:29:55,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:29:55,916 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:25,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:25,103 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:25,108 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:31:25,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:25,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1931, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:31:25,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:25,111 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1931, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:31:54,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:31:54,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:31:54,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.68 seconds 2025-02-15 14:31:54,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:54,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41377.55 MB 2025-02-15 14:31:54,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48211.26 MB 2025-02-15 14:31:54,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6833.70 MB 2025-02-15 14:31:54,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67480.06 MB 2025-02-15 14:31:54,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63759.71 MB 2025-02-15 14:31:54,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3720.35 MB 2025-02-15 14:31:54,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57190.71 MB 2025-02-15 14:31:54,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:31:54,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:31:54,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 14:31:54,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:54,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48211.26 MB 2025-02-15 14:31:54,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40769.85 MB 2025-02-15 14:31:54,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7441.41 MB 2025-02-15 14:31:54,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63759.71 MB 2025-02-15 14:31:54,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76908.86 MB 2025-02-15 14:31:54,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13149.14 MB 2025-02-15 14:31:54,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67986.11 MB 2025-02-15 14:31:56,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:31:56,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:31:56,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:31:56,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:56,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40769.85 MB 2025-02-15 14:31:56,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41300.69 MB 2025-02-15 14:31:56,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:31:56,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76908.86 MB 2025-02-15 14:31:56,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 14:31:56,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17320.38 MB 2025-02-15 14:31:56,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45279.24 MB 2025-02-15 14:31:56,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:31:56,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:31:56,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:31:56,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:56,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41300.69 MB 2025-02-15 14:31:56,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43190.18 MB 2025-02-15 14:31:56,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:31:56,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 14:31:56,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 14:31:56,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:56,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44607.61 MB 2025-02-15 14:31:57,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:31:57,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:31:57,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:31:57,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43190.18 MB 2025-02-15 14:31:57,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45432.04 MB 2025-02-15 14:31:57,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:31:57,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 14:31:57,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 14:31:57,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:57,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50976.32 MB 2025-02-15 14:31:57,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:31:57,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:31:57,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:31:57,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41300.69 MB 2025-02-15 14:31:57,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45432.04 MB 2025-02-15 14:31:57,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:31:57,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 14:31:57,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 14:31:57,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:57,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50976.32 MB 2025-02-15 14:31:57,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:31:57,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:31:57,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:31:57,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46139.83 MB 2025-02-15 14:31:57,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46906.83 MB 2025-02-15 14:31:57,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:31:57,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 14:31:57,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60003.71 MB 2025-02-15 14:31:57,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 14:31:57,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47614.62 MB 2025-02-15 14:31:57,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:31:57,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:31:57,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:31:57,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47319.72 MB 2025-02-15 14:31:57,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47526.29 MB 2025-02-15 14:31:57,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.57 MB 2025-02-15 14:31:57,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60003.71 MB 2025-02-15 14:31:57,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60003.71 MB 2025-02-15 14:31:57,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:57,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47726.97 MB 2025-02-15 14:31:57,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:31:57,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:31:57,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.14 seconds 2025-02-15 14:31:57,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34649.80 MB 2025-02-15 14:31:57,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47727.14 MB 2025-02-15 14:31:57,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13077.35 MB 2025-02-15 14:31:57,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67480.06 MB 2025-02-15 14:31:57,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60003.71 MB 2025-02-15 14:31:57,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7476.35 MB 2025-02-15 14:31:57,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47727.14 MB 2025-02-15 14:31:57,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:31:57,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:31:57,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:31:57,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47727.14 MB 2025-02-15 14:31:57,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47827.30 MB 2025-02-15 14:31:57,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.16 MB 2025-02-15 14:31:57,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60003.71 MB 2025-02-15 14:31:57,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60003.71 MB 2025-02-15 14:31:57,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:57,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48428.26 MB 2025-02-15 14:31:57,537 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 14:31:57,537 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:31:57,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:31:57,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:31:57,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:31:57,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:31:57,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35912.10 MB 2025-02-15 14:31:57,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40093.76 MB 2025-02-15 14:31:57,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.66 MB 2025-02-15 14:31:57,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60003.71 MB 2025-02-15 14:31:57,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60003.71 MB 2025-02-15 14:31:57,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:31:57,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44274.91 MB 2025-02-15 14:31:57,704 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 14:31:57,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:31:57,711 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:31:57,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,712 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:31:57,712 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:31:57,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,713 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,713 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:57,719 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:31:57,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,720 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,720 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:57,720 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:31:57,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,721 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:57,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,721 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:31:57,721 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:31:57,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,722 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:31:57,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,726 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,727 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,728 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:31:57,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:31:57,740 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:17,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:17,818 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:17,823 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:33:17,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:17,824 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1779, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:33:17,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:17,825 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1779, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:33:45,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:33:45,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:33:45,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.69 seconds 2025-02-15 14:33:45,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:45,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40440.12 MB 2025-02-15 14:33:45,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46735.90 MB 2025-02-15 14:33:45,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6295.78 MB 2025-02-15 14:33:45,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73987.52 MB 2025-02-15 14:33:45,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52388.95 MB 2025-02-15 14:33:45,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21598.57 MB 2025-02-15 14:33:45,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55573.80 MB 2025-02-15 14:33:45,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:33:45,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:33:45,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:33:45,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:45,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46735.90 MB 2025-02-15 14:33:45,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40102.42 MB 2025-02-15 14:33:45,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6633.48 MB 2025-02-15 14:33:45,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52388.95 MB 2025-02-15 14:33:45,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66364.38 MB 2025-02-15 14:33:45,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13975.42 MB 2025-02-15 14:33:45,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64557.10 MB 2025-02-15 14:33:47,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:33:47,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:33:47,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:33:47,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:47,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40102.42 MB 2025-02-15 14:33:47,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40633.26 MB 2025-02-15 14:33:47,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:33:47,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66364.38 MB 2025-02-15 14:33:47,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48215.62 MB 2025-02-15 14:33:47,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18148.75 MB 2025-02-15 14:33:47,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44611.81 MB 2025-02-15 14:33:47,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:33:47,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:33:47,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:33:47,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:47,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40633.26 MB 2025-02-15 14:33:47,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42522.76 MB 2025-02-15 14:33:47,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:33:47,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48215.62 MB 2025-02-15 14:33:47,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48215.62 MB 2025-02-15 14:33:47,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:33:47,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43940.19 MB 2025-02-15 14:33:47,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:33:47,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:33:47,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:33:47,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:47,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42522.76 MB 2025-02-15 14:33:47,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44764.61 MB 2025-02-15 14:33:47,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:33:47,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48215.62 MB 2025-02-15 14:33:47,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51990.50 MB 2025-02-15 14:33:47,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 14:33:47,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50308.89 MB 2025-02-15 14:33:47,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:33:47,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:33:47,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:33:47,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:47,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40633.26 MB 2025-02-15 14:33:47,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44764.61 MB 2025-02-15 14:33:47,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:33:47,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48215.62 MB 2025-02-15 14:33:47,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51990.50 MB 2025-02-15 14:33:47,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 14:33:47,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50308.89 MB 2025-02-15 14:33:47,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:33:47,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:33:47,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:33:47,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:47,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45472.40 MB 2025-02-15 14:33:47,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46239.40 MB 2025-02-15 14:33:47,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:33:47,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51990.50 MB 2025-02-15 14:33:47,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52407.83 MB 2025-02-15 14:33:47,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:33:47,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46947.19 MB 2025-02-15 14:33:48,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:33:48,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:33:48,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:33:48,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:48,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46652.29 MB 2025-02-15 14:33:48,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46858.20 MB 2025-02-15 14:33:48,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.91 MB 2025-02-15 14:33:48,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52407.83 MB 2025-02-15 14:33:48,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52407.83 MB 2025-02-15 14:33:48,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:33:48,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47070.54 MB 2025-02-15 14:33:48,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:33:48,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:33:48,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.18 seconds 2025-02-15 14:33:48,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:48,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34241.94 MB 2025-02-15 14:33:48,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47058.63 MB 2025-02-15 14:33:48,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12816.69 MB 2025-02-15 14:33:48,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73987.52 MB 2025-02-15 14:33:48,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52407.83 MB 2025-02-15 14:33:48,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21579.69 MB 2025-02-15 14:33:48,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47070.54 MB 2025-02-15 14:33:48,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:33:48,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:33:48,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:33:48,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:48,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47058.63 MB 2025-02-15 14:33:48,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47158.78 MB 2025-02-15 14:33:48,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.15 MB 2025-02-15 14:33:48,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52407.83 MB 2025-02-15 14:33:48,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52407.83 MB 2025-02-15 14:33:48,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:33:48,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47759.66 MB 2025-02-15 14:33:48,289 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-15 14:33:48,289 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:33:48,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:33:48,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:33:48,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:33:48,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:33:48,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35504.22 MB 2025-02-15 14:33:48,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39685.95 MB 2025-02-15 14:33:48,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.72 MB 2025-02-15 14:33:48,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52407.83 MB 2025-02-15 14:33:48,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60771.27 MB 2025-02-15 14:33:48,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-15 14:33:48,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43866.58 MB 2025-02-15 14:33:48,454 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-15 14:33:48,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,456 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,456 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:33:48,461 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:33:48,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,462 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:33:48,462 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:33:48,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,463 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,464 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:48,469 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:33:48,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,470 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,470 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:48,470 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:33:48,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,471 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:48,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,471 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:33:48,471 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:33:48,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,472 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:33:48,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,476 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,477 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,477 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:33:48,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:33:48,498 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:02,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:02,541 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:02,549 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:35:02,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:02,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1886, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:35:02,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:02,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1886, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:35:32,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:35:32,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:35:32,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.46 seconds 2025-02-15 14:35:32,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:32,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41307.44 MB 2025-02-15 14:35:32,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47982.67 MB 2025-02-15 14:35:32,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6675.23 MB 2025-02-15 14:35:32,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74876.72 MB 2025-02-15 14:35:32,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61249.42 MB 2025-02-15 14:35:32,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13627.29 MB 2025-02-15 14:35:32,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56894.11 MB 2025-02-15 14:35:32,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:35:32,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:35:32,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 14:35:32,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:32,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47982.67 MB 2025-02-15 14:35:32,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40779.36 MB 2025-02-15 14:35:32,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7203.31 MB 2025-02-15 14:35:32,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61249.42 MB 2025-02-15 14:35:32,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73922.51 MB 2025-02-15 14:35:32,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12673.09 MB 2025-02-15 14:35:32,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67038.29 MB 2025-02-15 14:35:34,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:35:34,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:35:34,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:35:34,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40779.36 MB 2025-02-15 14:35:34,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41310.20 MB 2025-02-15 14:35:34,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:35:34,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73922.51 MB 2025-02-15 14:35:34,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50392.47 MB 2025-02-15 14:35:34,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23530.05 MB 2025-02-15 14:35:34,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45288.75 MB 2025-02-15 14:35:34,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:35:34,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:35:34,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:35:34,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41310.20 MB 2025-02-15 14:35:34,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43199.69 MB 2025-02-15 14:35:34,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:35:34,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-15 14:35:34,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50392.47 MB 2025-02-15 14:35:34,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:35:34,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44617.12 MB 2025-02-15 14:35:34,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:35:34,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:35:34,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:35:34,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43199.69 MB 2025-02-15 14:35:34,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45441.55 MB 2025-02-15 14:35:34,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:35:34,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-15 14:35:34,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52751.76 MB 2025-02-15 14:35:34,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 14:35:34,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50985.83 MB 2025-02-15 14:35:34,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:35:34,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:35:34,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 14:35:34,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41310.20 MB 2025-02-15 14:35:34,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45441.55 MB 2025-02-15 14:35:34,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:35:34,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-15 14:35:34,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52751.76 MB 2025-02-15 14:35:34,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-15 14:35:34,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50985.83 MB 2025-02-15 14:35:34,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:35:34,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:35:34,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:35:34,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46149.34 MB 2025-02-15 14:35:34,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46916.34 MB 2025-02-15 14:35:34,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:35:34,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52751.76 MB 2025-02-15 14:35:34,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53169.09 MB 2025-02-15 14:35:34,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:35:34,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47624.13 MB 2025-02-15 14:35:34,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:35:34,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:35:34,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:35:34,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47329.23 MB 2025-02-15 14:35:34,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47536.16 MB 2025-02-15 14:35:34,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.93 MB 2025-02-15 14:35:34,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53169.09 MB 2025-02-15 14:35:34,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53169.09 MB 2025-02-15 14:35:34,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:35:34,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47748.66 MB 2025-02-15 14:35:34,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:35:34,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:35:34,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.01 seconds 2025-02-15 14:35:34,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34736.46 MB 2025-02-15 14:35:34,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47737.01 MB 2025-02-15 14:35:34,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13000.55 MB 2025-02-15 14:35:34,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74876.72 MB 2025-02-15 14:35:34,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53169.09 MB 2025-02-15 14:35:34,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21707.62 MB 2025-02-15 14:35:34,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47748.66 MB 2025-02-15 14:35:34,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:35:34,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:35:34,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:35:34,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47737.01 MB 2025-02-15 14:35:34,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47837.27 MB 2025-02-15 14:35:34,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-15 14:35:34,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53169.09 MB 2025-02-15 14:35:34,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53169.09 MB 2025-02-15 14:35:34,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:35:34,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48438.81 MB 2025-02-15 14:35:34,855 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 14:35:34,856 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:35:34,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:35:34,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:35:34,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:35:34,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:35:34,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35998.97 MB 2025-02-15 14:35:34,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40184.73 MB 2025-02-15 14:35:34,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.76 MB 2025-02-15 14:35:34,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53169.09 MB 2025-02-15 14:35:34,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57355.01 MB 2025-02-15 14:35:34,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 14:35:34,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44370.65 MB 2025-02-15 14:35:35,019 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 14:35:35,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,021 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,022 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:35:35,026 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:35:35,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,027 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:35:35,028 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:35:35,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,029 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,029 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:35,035 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:35:35,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,036 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,036 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:35,036 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:35:35,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,037 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:35,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,037 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:35:35,037 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:35:35,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,038 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:35:35,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,042 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,044 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,045 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:35:35,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:35:35,057 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:36:42,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:36:42,103 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:36:42,108 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:36:42,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:36:42,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1536, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:36:42,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:36:42,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1536, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:37:05,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:37:05,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:37:05,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.70 seconds 2025-02-15 14:37:05,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:05,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38990.31 MB 2025-02-15 14:37:05,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44426.13 MB 2025-02-15 14:37:05,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5435.82 MB 2025-02-15 14:37:05,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71582.09 MB 2025-02-15 14:37:05,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54700.02 MB 2025-02-15 14:37:05,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16882.07 MB 2025-02-15 14:37:05,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53444.51 MB 2025-02-15 14:37:05,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:37:05,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:37:05,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 14:37:05,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:05,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44426.13 MB 2025-02-15 14:37:05,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39081.54 MB 2025-02-15 14:37:05,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5344.58 MB 2025-02-15 14:37:05,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54700.02 MB 2025-02-15 14:37:05,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63803.75 MB 2025-02-15 14:37:05,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9103.74 MB 2025-02-15 14:37:05,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58117.13 MB 2025-02-15 14:37:07,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:37:07,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:37:07,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 14:37:07,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:07,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39081.54 MB 2025-02-15 14:37:07,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39612.39 MB 2025-02-15 14:37:07,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:37:07,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63803.75 MB 2025-02-15 14:37:07,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54700.02 MB 2025-02-15 14:37:07,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9103.74 MB 2025-02-15 14:37:07,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43590.93 MB 2025-02-15 14:37:07,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:37:07,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:37:07,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:37:07,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:07,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39612.39 MB 2025-02-15 14:37:07,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41501.88 MB 2025-02-15 14:37:07,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:37:07,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54700.02 MB 2025-02-15 14:37:07,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54700.02 MB 2025-02-15 14:37:07,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:37:07,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42919.31 MB 2025-02-15 14:37:08,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:37:08,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:37:08,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 14:37:08,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41501.88 MB 2025-02-15 14:37:08,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43743.73 MB 2025-02-15 14:37:08,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:37:08,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54700.02 MB 2025-02-15 14:37:08,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54702.11 MB 2025-02-15 14:37:08,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:37:08,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49288.02 MB 2025-02-15 14:37:08,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:37:08,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:37:08,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:37:08,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39612.39 MB 2025-02-15 14:37:08,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43743.73 MB 2025-02-15 14:37:08,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:37:08,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54700.02 MB 2025-02-15 14:37:08,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54702.11 MB 2025-02-15 14:37:08,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:37:08,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49288.02 MB 2025-02-15 14:37:08,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:37:08,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:37:08,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:37:08,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44451.52 MB 2025-02-15 14:37:08,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45218.52 MB 2025-02-15 14:37:08,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:37:08,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54702.11 MB 2025-02-15 14:37:08,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55119.45 MB 2025-02-15 14:37:08,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:37:08,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45926.31 MB 2025-02-15 14:37:08,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:37:08,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:37:08,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:37:08,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45631.41 MB 2025-02-15 14:37:08,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45837.32 MB 2025-02-15 14:37:08,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.91 MB 2025-02-15 14:37:08,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-15 14:37:08,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55119.45 MB 2025-02-15 14:37:08,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:37:08,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46046.69 MB 2025-02-15 14:37:08,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:37:08,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:37:08,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.19 seconds 2025-02-15 14:37:08,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33638.76 MB 2025-02-15 14:37:08,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46037.98 MB 2025-02-15 14:37:08,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12399.22 MB 2025-02-15 14:37:08,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71582.09 MB 2025-02-15 14:37:08,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55119.45 MB 2025-02-15 14:37:08,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16462.64 MB 2025-02-15 14:37:08,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46046.69 MB 2025-02-15 14:37:08,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:37:08,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:37:08,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 14:37:08,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46037.98 MB 2025-02-15 14:37:08,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46138.63 MB 2025-02-15 14:37:08,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.65 MB 2025-02-15 14:37:08,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-15 14:37:08,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55119.45 MB 2025-02-15 14:37:08,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:37:08,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46739.00 MB 2025-02-15 14:37:08,584 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-15 14:37:08,584 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:37:08,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:37:08,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:37:08,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:37:08,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:37:08,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34901.46 MB 2025-02-15 14:37:08,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39079.02 MB 2025-02-15 14:37:08,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.56 MB 2025-02-15 14:37:08,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-15 14:37:08,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55119.45 MB 2025-02-15 14:37:08,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:37:08,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43256.06 MB 2025-02-15 14:37:08,747 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-15 14:37:08,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,749 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,750 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:37:08,754 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:37:08,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,755 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:37:08,755 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:37:08,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,756 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,757 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:37:08,763 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:37:08,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,763 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,764 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:37:08,764 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:37:08,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,764 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:37:08,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,765 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:37:08,765 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:37:08,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,765 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:37:08,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,769 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,770 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,771 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:37:08,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:37:08,783 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:24,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:24,732 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:24,739 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:38:24,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:24,741 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1601, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:38:24,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:24,743 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1601, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:38:49,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:38:49,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:38:49,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.88 seconds 2025-02-15 14:38:49,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:49,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39564.96 MB 2025-02-15 14:38:49,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45230.81 MB 2025-02-15 14:38:49,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5665.85 MB 2025-02-15 14:38:49,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69468.16 MB 2025-02-15 14:38:49,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54821.65 MB 2025-02-15 14:38:49,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14646.51 MB 2025-02-15 14:38:49,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54245.66 MB 2025-02-15 14:38:49,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:38:49,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:38:49,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 14:38:49,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:49,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45230.81 MB 2025-02-15 14:38:49,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39541.18 MB 2025-02-15 14:38:49,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5689.63 MB 2025-02-15 14:38:49,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54821.65 MB 2025-02-15 14:38:49,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64349.01 MB 2025-02-15 14:38:49,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9527.36 MB 2025-02-15 14:38:49,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59555.79 MB 2025-02-15 14:38:51,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:38:51,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:38:51,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:38:51,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:51,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39541.18 MB 2025-02-15 14:38:51,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40072.03 MB 2025-02-15 14:38:51,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:38:51,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64349.01 MB 2025-02-15 14:38:51,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50639.93 MB 2025-02-15 14:38:51,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13709.08 MB 2025-02-15 14:38:51,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44050.57 MB 2025-02-15 14:38:51,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:38:51,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:38:51,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:38:51,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:51,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40072.03 MB 2025-02-15 14:38:51,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41961.52 MB 2025-02-15 14:38:51,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:38:51,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50639.93 MB 2025-02-15 14:38:51,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50639.93 MB 2025-02-15 14:38:51,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:38:51,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43378.95 MB 2025-02-15 14:38:51,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:38:51,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:38:51,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 14:38:51,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:51,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41961.52 MB 2025-02-15 14:38:51,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44203.38 MB 2025-02-15 14:38:51,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:38:51,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50639.93 MB 2025-02-15 14:38:51,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52055.51 MB 2025-02-15 14:38:51,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 14:38:51,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49747.66 MB 2025-02-15 14:38:51,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:38:51,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:38:51,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:38:51,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:51,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40072.03 MB 2025-02-15 14:38:51,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44203.38 MB 2025-02-15 14:38:51,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:38:51,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50639.93 MB 2025-02-15 14:38:51,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52055.51 MB 2025-02-15 14:38:51,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 14:38:51,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49747.66 MB 2025-02-15 14:38:52,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:38:52,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:38:52,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:38:52,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:52,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44911.16 MB 2025-02-15 14:38:52,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45678.17 MB 2025-02-15 14:38:52,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:38:52,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52055.51 MB 2025-02-15 14:38:52,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52472.84 MB 2025-02-15 14:38:52,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:38:52,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46385.95 MB 2025-02-15 14:38:52,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:38:52,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:38:52,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:38:52,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:52,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46091.05 MB 2025-02-15 14:38:52,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46296.63 MB 2025-02-15 14:38:52,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.58 MB 2025-02-15 14:38:52,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52472.84 MB 2025-02-15 14:38:52,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52472.84 MB 2025-02-15 14:38:52,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:38:52,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46523.17 MB 2025-02-15 14:38:52,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:38:52,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:38:52,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.31 seconds 2025-02-15 14:38:52,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:52,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33986.95 MB 2025-02-15 14:38:52,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46496.55 MB 2025-02-15 14:38:52,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12509.60 MB 2025-02-15 14:38:52,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69468.16 MB 2025-02-15 14:38:52,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52472.84 MB 2025-02-15 14:38:52,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16995.32 MB 2025-02-15 14:38:52,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46523.17 MB 2025-02-15 14:38:52,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:38:52,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:38:52,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:38:52,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:52,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46496.55 MB 2025-02-15 14:38:52,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46596.44 MB 2025-02-15 14:38:52,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.89 MB 2025-02-15 14:38:52,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52472.84 MB 2025-02-15 14:38:52,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52472.84 MB 2025-02-15 14:38:52,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:38:52,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47195.77 MB 2025-02-15 14:38:52,336 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 14:38:52,336 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:38:52,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:38:52,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:38:52,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 14:38:52,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:38:52,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35248.72 MB 2025-02-15 14:38:52,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39419.09 MB 2025-02-15 14:38:52,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4170.37 MB 2025-02-15 14:38:52,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52472.84 MB 2025-02-15 14:38:52,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56644.08 MB 2025-02-15 14:38:52,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 14:38:52,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43588.95 MB 2025-02-15 14:38:52,512 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 14:38:52,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,514 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:38:52,518 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:38:52,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,519 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:38:52,520 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:38:52,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,520 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,521 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:52,527 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:38:52,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,527 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,528 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:52,528 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:38:52,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,528 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:52,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,529 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:38:52,529 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:38:52,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,529 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:38:52,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,533 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,534 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,536 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:38:52,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:38:52,558 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:12,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:12,471 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:12,476 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:40:12,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:12,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1850, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:40:12,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:12,478 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1850, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:40:41,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:40:41,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:40:41,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.65 seconds 2025-02-15 14:40:41,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:41,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41421.76 MB 2025-02-15 14:40:41,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47968.81 MB 2025-02-15 14:40:41,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6547.05 MB 2025-02-15 14:40:41,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71114.42 MB 2025-02-15 14:40:41,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54932.80 MB 2025-02-15 14:40:41,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16181.62 MB 2025-02-15 14:40:41,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56781.94 MB 2025-02-15 14:40:41,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:40:41,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:40:41,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:40:41,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:41,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47968.81 MB 2025-02-15 14:40:41,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40957.38 MB 2025-02-15 14:40:41,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7011.43 MB 2025-02-15 14:40:41,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54932.80 MB 2025-02-15 14:40:41,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69126.32 MB 2025-02-15 14:40:41,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14193.52 MB 2025-02-15 14:40:41,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67183.47 MB 2025-02-15 14:40:43,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:40:43,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:40:43,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:40:43,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40957.38 MB 2025-02-15 14:40:43,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41488.23 MB 2025-02-15 14:40:43,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:40:43,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69126.32 MB 2025-02-15 14:40:43,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56348.38 MB 2025-02-15 14:40:43,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12777.95 MB 2025-02-15 14:40:43,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45466.77 MB 2025-02-15 14:40:43,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:40:43,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:40:43,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:40:43,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41488.23 MB 2025-02-15 14:40:43,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43377.72 MB 2025-02-15 14:40:43,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:40:43,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56348.38 MB 2025-02-15 14:40:43,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56348.38 MB 2025-02-15 14:40:43,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:40:43,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44795.15 MB 2025-02-15 14:40:43,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:40:43,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:40:43,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:40:43,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43377.72 MB 2025-02-15 14:40:43,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45619.57 MB 2025-02-15 14:40:43,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:40:43,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56348.38 MB 2025-02-15 14:40:43,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56350.47 MB 2025-02-15 14:40:43,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:40:43,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51163.86 MB 2025-02-15 14:40:43,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:40:43,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:40:43,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:40:43,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41488.23 MB 2025-02-15 14:40:43,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45619.57 MB 2025-02-15 14:40:43,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:40:43,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56348.38 MB 2025-02-15 14:40:43,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56350.47 MB 2025-02-15 14:40:43,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:40:43,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51163.86 MB 2025-02-15 14:40:43,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:40:43,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:40:43,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:40:43,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46327.36 MB 2025-02-15 14:40:43,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47094.36 MB 2025-02-15 14:40:43,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:40:43,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56350.47 MB 2025-02-15 14:40:43,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56767.81 MB 2025-02-15 14:40:43,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:40:43,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47802.15 MB 2025-02-15 14:40:43,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:40:43,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:40:43,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:40:43,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47507.25 MB 2025-02-15 14:40:43,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47713.02 MB 2025-02-15 14:40:43,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.76 MB 2025-02-15 14:40:43,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56767.81 MB 2025-02-15 14:40:43,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56767.81 MB 2025-02-15 14:40:43,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:40:43,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47927.98 MB 2025-02-15 14:40:43,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:40:43,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:40:43,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.14 seconds 2025-02-15 14:40:43,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34976.21 MB 2025-02-15 14:40:43,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47913.92 MB 2025-02-15 14:40:43,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12937.70 MB 2025-02-15 14:40:43,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71114.42 MB 2025-02-15 14:40:43,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56767.81 MB 2025-02-15 14:40:43,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14346.62 MB 2025-02-15 14:40:43,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47927.98 MB 2025-02-15 14:40:43,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:40:43,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:40:43,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:40:43,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47913.92 MB 2025-02-15 14:40:43,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48014.30 MB 2025-02-15 14:40:43,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.38 MB 2025-02-15 14:40:43,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56767.81 MB 2025-02-15 14:40:43,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56767.81 MB 2025-02-15 14:40:43,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:40:43,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48616.58 MB 2025-02-15 14:40:43,907 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 14:40:43,907 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:40:43,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:40:43,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:40:43,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:40:43,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:40:43,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36238.96 MB 2025-02-15 14:40:43,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40429.86 MB 2025-02-15 14:40:43,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.89 MB 2025-02-15 14:40:43,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56767.81 MB 2025-02-15 14:40:43,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56767.81 MB 2025-02-15 14:40:43,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:40:43,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44620.24 MB 2025-02-15 14:40:44,072 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 14:40:44,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:40:44,079 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:40:44,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,080 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:40:44,080 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:40:44,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,081 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,082 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:44,087 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:40:44,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,088 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,088 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:44,088 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:40:44,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,089 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:44,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,089 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:40:44,089 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:40:44,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,090 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:40:44,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,093 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,094 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,095 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:40:44,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:40:44,107 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:20,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:20,247 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:20,252 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:42:20,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:20,253 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1723, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:42:20,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:20,254 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1723, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:42:47,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:42:47,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:42:47,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.82 seconds 2025-02-15 14:42:47,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:47,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40658.53 MB 2025-02-15 14:42:47,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46756.13 MB 2025-02-15 14:42:47,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6097.60 MB 2025-02-15 14:42:47,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71359.79 MB 2025-02-15 14:42:47,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55054.43 MB 2025-02-15 14:42:47,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16305.36 MB 2025-02-15 14:42:47,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55565.72 MB 2025-02-15 14:42:47,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:42:47,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:42:47,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 14:42:47,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:47,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46756.13 MB 2025-02-15 14:42:47,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40418.88 MB 2025-02-15 14:42:47,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6337.26 MB 2025-02-15 14:42:47,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55054.43 MB 2025-02-15 14:42:47,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66616.03 MB 2025-02-15 14:42:47,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11561.60 MB 2025-02-15 14:42:47,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64144.80 MB 2025-02-15 14:42:49,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:42:49,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:42:49,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:42:49,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40418.88 MB 2025-02-15 14:42:49,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40949.72 MB 2025-02-15 14:42:49,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:42:49,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66616.03 MB 2025-02-15 14:42:49,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50883.20 MB 2025-02-15 14:42:49,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15732.83 MB 2025-02-15 14:42:49,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44928.27 MB 2025-02-15 14:42:49,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:42:49,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:42:49,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:42:49,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40949.72 MB 2025-02-15 14:42:49,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42838.89 MB 2025-02-15 14:42:49,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.18 MB 2025-02-15 14:42:49,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-15 14:42:49,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50885.30 MB 2025-02-15 14:42:49,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:42:49,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44256.32 MB 2025-02-15 14:42:49,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:42:49,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:42:49,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:42:49,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42838.89 MB 2025-02-15 14:42:49,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45080.75 MB 2025-02-15 14:42:49,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:42:49,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50885.30 MB 2025-02-15 14:42:49,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52772.73 MB 2025-02-15 14:42:49,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 14:42:49,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50625.03 MB 2025-02-15 14:42:49,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:42:49,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:42:49,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:42:49,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40949.72 MB 2025-02-15 14:42:49,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45080.75 MB 2025-02-15 14:42:49,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 14:42:49,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-15 14:42:49,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52772.73 MB 2025-02-15 14:42:49,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 14:42:49,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50625.03 MB 2025-02-15 14:42:49,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:42:49,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:42:49,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:42:49,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45788.54 MB 2025-02-15 14:42:49,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46555.54 MB 2025-02-15 14:42:49,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:42:49,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52772.73 MB 2025-02-15 14:42:49,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 14:42:49,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:42:49,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47263.33 MB 2025-02-15 14:42:49,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:42:49,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:42:49,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:42:49,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46968.43 MB 2025-02-15 14:42:49,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47173.59 MB 2025-02-15 14:42:49,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.16 MB 2025-02-15 14:42:49,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53190.07 MB 2025-02-15 14:42:49,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 14:42:49,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:42:49,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47372.66 MB 2025-02-15 14:42:49,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:42:49,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:42:49,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.29 seconds 2025-02-15 14:42:49,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34655.46 MB 2025-02-15 14:42:49,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47374.49 MB 2025-02-15 14:42:49,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12719.03 MB 2025-02-15 14:42:49,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71359.79 MB 2025-02-15 14:42:49,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 14:42:49,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18169.72 MB 2025-02-15 14:42:49,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47374.49 MB 2025-02-15 14:42:49,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:42:49,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:42:49,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:42:49,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47374.49 MB 2025-02-15 14:42:49,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47474.54 MB 2025-02-15 14:42:49,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.05 MB 2025-02-15 14:42:49,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53190.07 MB 2025-02-15 14:42:49,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 14:42:49,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:42:49,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48074.83 MB 2025-02-15 14:42:49,829 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 14:42:49,829 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:42:49,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:42:49,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:42:49,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:42:49,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:42:49,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35918.21 MB 2025-02-15 14:42:49,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40095.25 MB 2025-02-15 14:42:49,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.04 MB 2025-02-15 14:42:49,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53190.07 MB 2025-02-15 14:42:49,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53190.07 MB 2025-02-15 14:42:49,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:42:49,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44271.78 MB 2025-02-15 14:42:50,003 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 14:42:50,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,004 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,005 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:42:50,010 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:42:50,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,011 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:42:50,011 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:42:50,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,012 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,012 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:50,018 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:42:50,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,019 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,019 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:50,019 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:42:50,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,020 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:50,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,020 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:42:50,020 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:42:50,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,021 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:42:50,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,024 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,025 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,026 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:42:50,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:42:50,038 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:18,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:18,914 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:18,919 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:44:18,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:18,920 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2076, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:44:18,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:18,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2076, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:44:51,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:44:51,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:44:51,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.09 seconds 2025-02-15 14:44:51,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:51,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43240.99 MB 2025-02-15 14:44:51,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50587.84 MB 2025-02-15 14:44:51,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7346.85 MB 2025-02-15 14:44:51,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67903.68 MB 2025-02-15 14:44:51,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59204.70 MB 2025-02-15 14:44:51,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8698.99 MB 2025-02-15 14:44:51,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59507.13 MB 2025-02-15 14:44:51,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:44:51,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:44:51,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:44:51,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:51,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50587.84 MB 2025-02-15 14:44:51,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42376.71 MB 2025-02-15 14:44:51,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8211.13 MB 2025-02-15 14:44:51,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59204.70 MB 2025-02-15 14:44:51,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73253.52 MB 2025-02-15 14:44:51,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14048.82 MB 2025-02-15 14:44:51,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 71716.26 MB 2025-02-15 14:44:53,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:44:53,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:44:53,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:44:53,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42376.71 MB 2025-02-15 14:44:53,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42907.55 MB 2025-02-15 14:44:53,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:44:53,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73253.52 MB 2025-02-15 14:44:53,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51856.28 MB 2025-02-15 14:44:53,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21397.24 MB 2025-02-15 14:44:53,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46886.10 MB 2025-02-15 14:44:53,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:44:53,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:44:53,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:44:53,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42907.55 MB 2025-02-15 14:44:53,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44797.04 MB 2025-02-15 14:44:53,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 14:44:53,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51856.28 MB 2025-02-15 14:44:53,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51858.37 MB 2025-02-15 14:44:53,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:44:53,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46214.47 MB 2025-02-15 14:44:53,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:44:53,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:44:53,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:44:53,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44797.04 MB 2025-02-15 14:44:53,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47038.90 MB 2025-02-15 14:44:53,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:44:53,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51858.37 MB 2025-02-15 14:44:53,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54689.53 MB 2025-02-15 14:44:53,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 14:44:53,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52583.18 MB 2025-02-15 14:44:53,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:44:53,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:44:53,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:44:53,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42907.55 MB 2025-02-15 14:44:53,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47038.90 MB 2025-02-15 14:44:53,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 14:44:53,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51856.28 MB 2025-02-15 14:44:53,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54689.53 MB 2025-02-15 14:44:53,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 14:44:53,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52583.18 MB 2025-02-15 14:44:53,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:44:53,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:44:53,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 14:44:53,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47746.69 MB 2025-02-15 14:44:53,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48513.69 MB 2025-02-15 14:44:53,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:44:53,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54689.53 MB 2025-02-15 14:44:53,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55106.86 MB 2025-02-15 14:44:53,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:44:53,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49221.48 MB 2025-02-15 14:44:53,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:44:53,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:44:53,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:44:53,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48926.58 MB 2025-02-15 14:44:53,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49133.42 MB 2025-02-15 14:44:53,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.84 MB 2025-02-15 14:44:53,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55106.86 MB 2025-02-15 14:44:53,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55106.86 MB 2025-02-15 14:44:53,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:44:53,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49340.96 MB 2025-02-15 14:44:53,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:44:53,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:44:53,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.59 seconds 2025-02-15 14:44:53,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36008.04 MB 2025-02-15 14:44:53,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49334.47 MB 2025-02-15 14:44:53,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13326.43 MB 2025-02-15 14:44:53,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67903.68 MB 2025-02-15 14:44:53,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55106.86 MB 2025-02-15 14:44:53,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12796.82 MB 2025-02-15 14:44:53,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49340.96 MB 2025-02-15 14:44:53,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:44:53,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:44:53,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:44:53,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49334.47 MB 2025-02-15 14:44:53,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49434.92 MB 2025-02-15 14:44:53,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.45 MB 2025-02-15 14:44:53,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55106.86 MB 2025-02-15 14:44:53,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55106.86 MB 2025-02-15 14:44:53,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:44:53,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50037.65 MB 2025-02-15 14:44:53,800 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 14:44:53,800 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:44:53,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:44:53,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:44:53,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:44:53,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:44:53,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37270.94 MB 2025-02-15 14:44:53,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41464.91 MB 2025-02-15 14:44:53,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4193.97 MB 2025-02-15 14:44:53,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55106.86 MB 2025-02-15 14:44:53,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55106.86 MB 2025-02-15 14:44:53,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:44:53,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45658.37 MB 2025-02-15 14:44:53,965 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 14:44:53,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,966 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:53,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:44:53,972 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:44:53,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:44:53,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:44:53,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,974 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:53,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,974 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:53,980 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:44:53,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,981 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:53,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,981 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:53,981 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:44:53,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,982 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:53,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,982 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:44:53,982 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:44:53,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,983 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:44:53,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,986 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:53,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,987 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:53,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:53,988 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:44:54,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:44:54,001 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:46:38,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:46:38,717 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:46:38,722 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:46:38,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:46:38,723 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1814, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:46:38,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:46:38,724 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1814, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:47:06,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:47:06,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:47:06,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.82 seconds 2025-02-15 14:47:06,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:06,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41536.99 MB 2025-02-15 14:47:06,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47956.64 MB 2025-02-15 14:47:06,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6419.64 MB 2025-02-15 14:47:06,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69942.12 MB 2025-02-15 14:47:06,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66691.53 MB 2025-02-15 14:47:06,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3250.59 MB 2025-02-15 14:47:06,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56897.17 MB 2025-02-15 14:47:06,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:47:06,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:47:06,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 14:47:06,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:06,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47956.64 MB 2025-02-15 14:47:06,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41136.31 MB 2025-02-15 14:47:06,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6820.32 MB 2025-02-15 14:47:06,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66691.53 MB 2025-02-15 14:47:06,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72932.66 MB 2025-02-15 14:47:06,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6241.12 MB 2025-02-15 14:47:06,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66577.44 MB 2025-02-15 14:47:08,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:47:08,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:47:08,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:47:08,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41136.31 MB 2025-02-15 14:47:08,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41667.16 MB 2025-02-15 14:47:08,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:47:08,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72932.66 MB 2025-02-15 14:47:08,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62505.62 MB 2025-02-15 14:47:08,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10427.04 MB 2025-02-15 14:47:08,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45645.70 MB 2025-02-15 14:47:08,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:47:08,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:47:08,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:47:08,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41667.16 MB 2025-02-15 14:47:08,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43556.27 MB 2025-02-15 14:47:08,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.11 MB 2025-02-15 14:47:08,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 14:47:08,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62505.62 MB 2025-02-15 14:47:08,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:08,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44973.70 MB 2025-02-15 14:47:08,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:47:08,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:47:08,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:47:08,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43556.27 MB 2025-02-15 14:47:08,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45798.13 MB 2025-02-15 14:47:08,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:47:08,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 14:47:08,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62505.62 MB 2025-02-15 14:47:08,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:08,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51342.41 MB 2025-02-15 14:47:08,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:47:08,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:47:08,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:47:08,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41667.16 MB 2025-02-15 14:47:08,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45798.13 MB 2025-02-15 14:47:08,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.97 MB 2025-02-15 14:47:08,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 14:47:08,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62505.62 MB 2025-02-15 14:47:08,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:08,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51342.41 MB 2025-02-15 14:47:08,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:47:08,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:47:08,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:47:08,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46505.91 MB 2025-02-15 14:47:08,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47272.92 MB 2025-02-15 14:47:08,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:47:08,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62505.62 MB 2025-02-15 14:47:08,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62922.95 MB 2025-02-15 14:47:08,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:47:08,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47980.71 MB 2025-02-15 14:47:08,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:47:08,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:47:08,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:47:08,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47685.81 MB 2025-02-15 14:47:08,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47893.09 MB 2025-02-15 14:47:08,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.28 MB 2025-02-15 14:47:08,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62922.95 MB 2025-02-15 14:47:08,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62922.95 MB 2025-02-15 14:47:08,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:08,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48101.81 MB 2025-02-15 14:47:08,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:47:08,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:47:08,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.27 seconds 2025-02-15 14:47:08,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:08,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35216.87 MB 2025-02-15 14:47:08,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48094.16 MB 2025-02-15 14:47:08,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12877.29 MB 2025-02-15 14:47:08,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69942.12 MB 2025-02-15 14:47:08,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62922.95 MB 2025-02-15 14:47:08,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7019.17 MB 2025-02-15 14:47:08,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48101.81 MB 2025-02-15 14:47:09,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:47:09,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:47:09,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:47:09,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:09,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48094.16 MB 2025-02-15 14:47:09,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48194.63 MB 2025-02-15 14:47:09,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 14:47:09,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62922.95 MB 2025-02-15 14:47:09,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62922.95 MB 2025-02-15 14:47:09,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:09,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48797.43 MB 2025-02-15 14:47:09,283 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 14:47:09,283 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:47:09,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:47:09,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:47:09,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:47:09,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:47:09,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36479.79 MB 2025-02-15 14:47:09,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40674.28 MB 2025-02-15 14:47:09,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 14:47:09,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62922.95 MB 2025-02-15 14:47:09,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62922.95 MB 2025-02-15 14:47:09,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:47:09,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44868.25 MB 2025-02-15 14:47:09,448 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 14:47:09,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,449 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,450 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:47:09,455 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:47:09,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,456 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:47:09,456 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:47:09,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,457 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,457 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:47:09,463 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:47:09,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,463 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,464 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:47:09,464 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:47:09,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,464 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:47:09,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,465 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:47:09,465 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:47:09,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,465 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:47:09,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,469 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,470 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,471 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:47:09,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:47:09,495 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:48:40,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:48:40,703 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:48:40,708 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:48:40,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:48:40,709 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2521, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:48:40,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:48:40,710 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2521, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:49:19,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:49:19,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:49:19,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.12 seconds 2025-02-15 14:49:19,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:19,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46584.30 MB 2025-02-15 14:49:19,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55505.98 MB 2025-02-15 14:49:19,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8921.68 MB 2025-02-15 14:49:19,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77879.84 MB 2025-02-15 14:49:19,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71550.63 MB 2025-02-15 14:49:19,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6329.20 MB 2025-02-15 14:49:19,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64435.89 MB 2025-02-15 14:49:20,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:49:20,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:49:20,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 14:49:20,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:20,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 55505.98 MB 2025-02-15 14:49:20,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44932.61 MB 2025-02-15 14:49:20,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10573.37 MB 2025-02-15 14:49:20,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71550.63 MB 2025-02-15 14:49:20,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 88571.12 MB 2025-02-15 14:49:20,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17020.49 MB 2025-02-15 14:49:20,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 81098.77 MB 2025-02-15 14:49:21,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:49:21,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:49:21,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:49:21,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:21,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44932.61 MB 2025-02-15 14:49:21,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45463.45 MB 2025-02-15 14:49:21,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:49:21,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 88571.12 MB 2025-02-15 14:49:21,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62627.25 MB 2025-02-15 14:49:21,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25943.87 MB 2025-02-15 14:49:21,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49442.00 MB 2025-02-15 14:49:21,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:49:21,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:49:21,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:49:21,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:21,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45463.45 MB 2025-02-15 14:49:21,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47352.61 MB 2025-02-15 14:49:21,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.16 MB 2025-02-15 14:49:21,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62627.25 MB 2025-02-15 14:49:21,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62627.25 MB 2025-02-15 14:49:21,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:49:21,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48770.04 MB 2025-02-15 14:49:22,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:49:22,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:49:22,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:49:22,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47352.61 MB 2025-02-15 14:49:22,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49594.47 MB 2025-02-15 14:49:22,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:49:22,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62627.25 MB 2025-02-15 14:49:22,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62629.35 MB 2025-02-15 14:49:22,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:49:22,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55138.75 MB 2025-02-15 14:49:22,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:49:22,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:49:22,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:49:22,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45463.45 MB 2025-02-15 14:49:22,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49594.47 MB 2025-02-15 14:49:22,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.02 MB 2025-02-15 14:49:22,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62627.25 MB 2025-02-15 14:49:22,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62629.35 MB 2025-02-15 14:49:22,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:49:22,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55138.75 MB 2025-02-15 14:49:22,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:49:22,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:49:22,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:49:22,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50302.26 MB 2025-02-15 14:49:22,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51069.26 MB 2025-02-15 14:49:22,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:49:22,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62629.35 MB 2025-02-15 14:49:22,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63046.68 MB 2025-02-15 14:49:22,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:49:22,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51777.05 MB 2025-02-15 14:49:22,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:49:22,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:49:22,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:49:22,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51482.15 MB 2025-02-15 14:49:22,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51687.92 MB 2025-02-15 14:49:22,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.77 MB 2025-02-15 14:49:22,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63046.68 MB 2025-02-15 14:49:22,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63046.68 MB 2025-02-15 14:49:22,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:49:22,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51887.49 MB 2025-02-15 14:49:22,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:49:22,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:49:22,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.71 seconds 2025-02-15 14:49:22,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37800.94 MB 2025-02-15 14:49:22,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51888.77 MB 2025-02-15 14:49:22,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14087.83 MB 2025-02-15 14:49:22,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77879.84 MB 2025-02-15 14:49:22,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63046.68 MB 2025-02-15 14:49:22,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14833.16 MB 2025-02-15 14:49:22,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51888.77 MB 2025-02-15 14:49:22,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:49:22,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:49:22,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:49:22,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51888.77 MB 2025-02-15 14:49:22,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51989.73 MB 2025-02-15 14:49:22,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.96 MB 2025-02-15 14:49:22,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63046.68 MB 2025-02-15 14:49:22,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63046.68 MB 2025-02-15 14:49:22,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:49:22,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52589.58 MB 2025-02-15 14:49:22,703 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-15 14:49:22,704 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:49:22,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:49:22,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:49:22,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:49:22,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:49:22,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39063.86 MB 2025-02-15 14:49:22,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43237.82 MB 2025-02-15 14:49:22,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.96 MB 2025-02-15 14:49:22,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63046.68 MB 2025-02-15 14:49:22,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63046.68 MB 2025-02-15 14:49:22,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:49:22,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47411.27 MB 2025-02-15 14:49:22,868 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-15 14:49:22,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,869 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,870 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:49:22,875 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:49:22,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,876 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:49:22,876 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:49:22,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,877 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,877 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:49:22,883 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:49:22,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,883 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,884 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:49:22,884 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:49:22,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,884 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:49:22,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,885 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:49:22,885 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:49:22,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,885 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:49:22,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,889 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,890 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,891 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:49:22,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:49:22,905 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:51:27,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:51:27,636 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:51:27,641 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:51:27,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:51:27,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1921, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:51:27,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:51:27,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1921, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:51:57,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:51:57,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:51:57,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.52 seconds 2025-02-15 14:51:57,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:57,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42525.91 MB 2025-02-15 14:51:57,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49324.22 MB 2025-02-15 14:51:57,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6798.31 MB 2025-02-15 14:51:57,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78125.20 MB 2025-02-15 14:51:57,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 14:51:57,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15376.32 MB 2025-02-15 14:51:57,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58339.06 MB 2025-02-15 14:51:57,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:51:57,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:51:57,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 14:51:57,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:57,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49324.22 MB 2025-02-15 14:51:57,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41935.89 MB 2025-02-15 14:51:57,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7388.32 MB 2025-02-15 14:51:57,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 14:51:57,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75822.53 MB 2025-02-15 14:51:57,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13073.65 MB 2025-02-15 14:51:57,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68982.71 MB 2025-02-15 14:51:59,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:51:59,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:51:59,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 14:51:59,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41935.89 MB 2025-02-15 14:51:59,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42466.74 MB 2025-02-15 14:51:59,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:51:59,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75822.53 MB 2025-02-15 14:51:59,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 14:51:59,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13073.65 MB 2025-02-15 14:51:59,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46445.28 MB 2025-02-15 14:51:59,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:51:59,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:51:59,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:51:59,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42466.74 MB 2025-02-15 14:51:59,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44355.95 MB 2025-02-15 14:51:59,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-15 14:51:59,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 14:51:59,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 14:51:59,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45773.38 MB 2025-02-15 14:51:59,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:51:59,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:51:59,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:51:59,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44355.95 MB 2025-02-15 14:51:59,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46597.80 MB 2025-02-15 14:51:59,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:51:59,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 14:51:59,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 14:51:59,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52142.08 MB 2025-02-15 14:51:59,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:51:59,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:51:59,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:51:59,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42466.74 MB 2025-02-15 14:51:59,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46597.80 MB 2025-02-15 14:51:59,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.07 MB 2025-02-15 14:51:59,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 14:51:59,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62748.88 MB 2025-02-15 14:51:59,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52142.08 MB 2025-02-15 14:51:59,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:51:59,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:51:59,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:51:59,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47305.59 MB 2025-02-15 14:51:59,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48072.59 MB 2025-02-15 14:51:59,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:51:59,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62748.88 MB 2025-02-15 14:51:59,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63166.22 MB 2025-02-15 14:51:59,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:51:59,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48780.38 MB 2025-02-15 14:51:59,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:51:59,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:51:59,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:51:59,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48485.48 MB 2025-02-15 14:51:59,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48691.89 MB 2025-02-15 14:51:59,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.41 MB 2025-02-15 14:51:59,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63166.22 MB 2025-02-15 14:51:59,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63166.22 MB 2025-02-15 14:51:59,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48898.41 MB 2025-02-15 14:51:59,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:51:59,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:51:59,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.01 seconds 2025-02-15 14:51:59,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35832.99 MB 2025-02-15 14:51:59,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48892.74 MB 2025-02-15 14:51:59,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13059.75 MB 2025-02-15 14:51:59,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78125.20 MB 2025-02-15 14:51:59,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63166.22 MB 2025-02-15 14:51:59,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14958.99 MB 2025-02-15 14:51:59,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48898.41 MB 2025-02-15 14:51:59,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:51:59,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:51:59,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:51:59,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48892.74 MB 2025-02-15 14:51:59,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48992.79 MB 2025-02-15 14:51:59,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.05 MB 2025-02-15 14:51:59,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63166.22 MB 2025-02-15 14:51:59,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63166.22 MB 2025-02-15 14:51:59,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49593.08 MB 2025-02-15 14:51:59,937 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 14:51:59,937 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:51:59,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:51:59,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:51:59,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:51:59,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:51:59,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37095.07 MB 2025-02-15 14:51:59,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41272.12 MB 2025-02-15 14:51:59,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4177.04 MB 2025-02-15 14:51:59,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63166.22 MB 2025-02-15 14:51:59,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63166.22 MB 2025-02-15 14:51:59,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:51:59,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45448.64 MB 2025-02-15 14:52:00,102 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 14:52:00,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,104 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,105 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:52:00,109 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:52:00,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,110 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:52:00,111 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:52:00,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,111 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,112 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:52:00,118 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:52:00,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,118 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,119 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:52:00,119 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:52:00,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,119 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:52:00,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,120 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:52:00,120 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:52:00,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,120 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:52:00,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,124 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,125 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,125 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:52:00,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:52:00,148 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:53:24,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:53:24,899 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:53:24,904 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:53:24,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:53:24,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2627, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:53:24,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:53:24,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2627, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:54:05,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:54:05,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:54:05,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.56 seconds 2025-02-15 14:54:05,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:05,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47567.79 MB 2025-02-15 14:54:05,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 56864.60 MB 2025-02-15 14:54:05,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9296.81 MB 2025-02-15 14:54:05,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78366.38 MB 2025-02-15 14:54:05,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63235.42 MB 2025-02-15 14:54:05,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15130.95 MB 2025-02-15 14:54:05,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66161.40 MB 2025-02-15 14:54:05,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:54:05,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:54:05,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-15 14:54:05,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:05,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 56864.60 MB 2025-02-15 14:54:05,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45728.53 MB 2025-02-15 14:54:05,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11136.06 MB 2025-02-15 14:54:05,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63235.42 MB 2025-02-15 14:54:05,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 82128.67 MB 2025-02-15 14:54:05,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18893.24 MB 2025-02-15 14:54:05,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 83147.40 MB 2025-02-15 14:54:07,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:54:07,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:54:07,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 14:54:07,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:07,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45728.53 MB 2025-02-15 14:54:07,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46259.38 MB 2025-02-15 14:54:07,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:54:07,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82128.67 MB 2025-02-15 14:54:07,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64651.00 MB 2025-02-15 14:54:07,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17477.66 MB 2025-02-15 14:54:07,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50237.92 MB 2025-02-15 14:54:07,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:54:07,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:54:07,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:54:07,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:07,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46259.38 MB 2025-02-15 14:54:07,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48148.63 MB 2025-02-15 14:54:07,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.26 MB 2025-02-15 14:54:07,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64651.00 MB 2025-02-15 14:54:07,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64651.00 MB 2025-02-15 14:54:07,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:54:07,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49566.06 MB 2025-02-15 14:54:07,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:54:07,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:54:07,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:54:07,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:07,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48148.63 MB 2025-02-15 14:54:07,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50390.49 MB 2025-02-15 14:54:07,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:54:07,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64651.00 MB 2025-02-15 14:54:07,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64653.10 MB 2025-02-15 14:54:07,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:54:07,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55934.77 MB 2025-02-15 14:54:07,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:54:07,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:54:07,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:54:07,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:07,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46259.38 MB 2025-02-15 14:54:07,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50390.49 MB 2025-02-15 14:54:07,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.11 MB 2025-02-15 14:54:07,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64651.00 MB 2025-02-15 14:54:07,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64653.10 MB 2025-02-15 14:54:07,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 14:54:07,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55934.77 MB 2025-02-15 14:54:08,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:54:08,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:54:08,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 14:54:08,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:08,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51098.28 MB 2025-02-15 14:54:08,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51865.28 MB 2025-02-15 14:54:08,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:54:08,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64653.10 MB 2025-02-15 14:54:08,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65070.43 MB 2025-02-15 14:54:08,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:54:08,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52573.07 MB 2025-02-15 14:54:08,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:54:08,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:54:08,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:54:08,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:08,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52278.17 MB 2025-02-15 14:54:08,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52484.55 MB 2025-02-15 14:54:08,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.38 MB 2025-02-15 14:54:08,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65070.43 MB 2025-02-15 14:54:08,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65070.43 MB 2025-02-15 14:54:08,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:54:08,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.29 MB 2025-02-15 14:54:08,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:54:08,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:54:08,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.29 seconds 2025-02-15 14:54:08,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:08,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38415.11 MB 2025-02-15 14:54:08,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52685.40 MB 2025-02-15 14:54:08,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14270.29 MB 2025-02-15 14:54:08,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78366.38 MB 2025-02-15 14:54:08,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65070.43 MB 2025-02-15 14:54:08,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13295.94 MB 2025-02-15 14:54:08,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.29 MB 2025-02-15 14:54:08,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:54:08,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:54:08,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:54:08,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:08,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52685.40 MB 2025-02-15 14:54:08,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52785.70 MB 2025-02-15 14:54:08,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.29 MB 2025-02-15 14:54:08,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65070.43 MB 2025-02-15 14:54:08,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65070.43 MB 2025-02-15 14:54:08,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:54:08,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53387.47 MB 2025-02-15 14:54:08,483 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 14:54:08,484 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:54:08,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:54:08,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:54:08,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:54:08,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:54:08,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39677.81 MB 2025-02-15 14:54:08,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43865.12 MB 2025-02-15 14:54:08,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4187.30 MB 2025-02-15 14:54:08,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65070.43 MB 2025-02-15 14:54:08,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65070.43 MB 2025-02-15 14:54:08,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:54:08,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48051.91 MB 2025-02-15 14:54:08,647 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 14:54:08,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,648 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,649 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:54:08,653 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:54:08,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,654 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:54:08,655 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:54:08,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,655 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,656 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:54:08,662 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:54:08,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,662 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,663 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:54:08,663 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:54:08,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,663 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:54:08,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,664 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:54:08,664 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:54:08,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,664 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:54:08,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,669 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,671 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,672 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:54:08,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:54:08,686 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:55:34,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:55:34,748 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:55:34,752 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:55:34,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:55:34,754 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1949, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:55:34,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:55:34,755 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1949, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:56:05,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:56:05,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:56:05,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.27 seconds 2025-02-15 14:56:05,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:05,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42963.69 MB 2025-02-15 14:56:05,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49861.23 MB 2025-02-15 14:56:05,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6897.53 MB 2025-02-15 14:56:05,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80392.22 MB 2025-02-15 14:56:05,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55662.61 MB 2025-02-15 14:56:05,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24729.62 MB 2025-02-15 14:56:05,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58776.85 MB 2025-02-15 14:56:05,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:56:05,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:56:05,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 14:56:05,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:05,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49861.23 MB 2025-02-15 14:56:05,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42325.18 MB 2025-02-15 14:56:05,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7536.04 MB 2025-02-15 14:56:05,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55662.61 MB 2025-02-15 14:56:05,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70921.49 MB 2025-02-15 14:56:05,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15258.88 MB 2025-02-15 14:56:05,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69616.43 MB 2025-02-15 14:56:07,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:56:07,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:56:07,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 14:56:07,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42325.18 MB 2025-02-15 14:56:07,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42856.03 MB 2025-02-15 14:56:07,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:56:07,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70921.49 MB 2025-02-15 14:56:07,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50889.49 MB 2025-02-15 14:56:07,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20032.00 MB 2025-02-15 14:56:07,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46834.57 MB 2025-02-15 14:56:07,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:56:07,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:56:07,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:56:07,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42856.03 MB 2025-02-15 14:56:07,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44745.33 MB 2025-02-15 14:56:07,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.31 MB 2025-02-15 14:56:07,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50889.49 MB 2025-02-15 14:56:07,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50889.49 MB 2025-02-15 14:56:07,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:56:07,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46162.76 MB 2025-02-15 14:56:07,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:56:07,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:56:07,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:56:07,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44745.33 MB 2025-02-15 14:56:07,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46987.19 MB 2025-02-15 14:56:07,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:56:07,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50889.49 MB 2025-02-15 14:56:07,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54192.50 MB 2025-02-15 14:56:07,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 14:56:07,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52531.47 MB 2025-02-15 14:56:07,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:56:07,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:56:07,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:56:07,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42856.03 MB 2025-02-15 14:56:07,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46987.19 MB 2025-02-15 14:56:07,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.16 MB 2025-02-15 14:56:07,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50889.49 MB 2025-02-15 14:56:07,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54192.50 MB 2025-02-15 14:56:07,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-15 14:56:07,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52531.47 MB 2025-02-15 14:56:07,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:56:07,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:56:07,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:56:07,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47694.98 MB 2025-02-15 14:56:07,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48461.98 MB 2025-02-15 14:56:07,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:56:07,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54192.50 MB 2025-02-15 14:56:07,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54609.84 MB 2025-02-15 14:56:07,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:56:07,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49169.77 MB 2025-02-15 14:56:07,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:56:07,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:56:07,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:56:07,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48874.87 MB 2025-02-15 14:56:07,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49081.58 MB 2025-02-15 14:56:07,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.72 MB 2025-02-15 14:56:07,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54609.84 MB 2025-02-15 14:56:07,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54609.84 MB 2025-02-15 14:56:07,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:56:07,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49280.38 MB 2025-02-15 14:56:07,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:56:07,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:56:07,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.78 seconds 2025-02-15 14:56:07,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36173.22 MB 2025-02-15 14:56:07,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49282.46 MB 2025-02-15 14:56:07,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13109.24 MB 2025-02-15 14:56:07,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80392.22 MB 2025-02-15 14:56:07,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54609.84 MB 2025-02-15 14:56:07,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25782.39 MB 2025-02-15 14:56:07,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49282.46 MB 2025-02-15 14:56:07,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:56:07,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:56:07,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:56:07,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49282.46 MB 2025-02-15 14:56:07,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49382.83 MB 2025-02-15 14:56:07,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.37 MB 2025-02-15 14:56:07,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54609.84 MB 2025-02-15 14:56:07,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54609.84 MB 2025-02-15 14:56:07,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:56:07,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49985.04 MB 2025-02-15 14:56:07,822 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 14:56:07,822 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:56:07,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:56:07,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:56:07,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:56:07,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:56:07,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37435.94 MB 2025-02-15 14:56:07,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41626.33 MB 2025-02-15 14:56:07,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.38 MB 2025-02-15 14:56:07,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54609.84 MB 2025-02-15 14:56:07,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58799.95 MB 2025-02-15 14:56:07,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-15 14:56:07,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45816.44 MB 2025-02-15 14:56:07,985 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 14:56:07,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:07,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:07,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:07,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:56:07,992 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:56:07,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:07,993 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:56:07,993 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 14:56:07,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:07,994 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:07,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:07,995 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:56:08,000 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:56:08,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,001 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:08,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,001 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:56:08,001 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:56:08,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,002 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:56:08,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,002 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:56:08,002 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:56:08,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,003 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:56:08,006 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,006 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:08,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,007 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:08,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,008 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:56:08,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:56:08,021 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:57:39,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:57:39,473 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:57:39,478 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 14:57:39,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:57:39,479 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1831, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 14:57:39,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:57:39,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1831, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 14:58:07,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 14:58:07,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 14:58:07,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.39 seconds 2025-02-15 14:58:07,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:07,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42264.57 MB 2025-02-15 14:58:07,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48744.77 MB 2025-02-15 14:58:07,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6480.20 MB 2025-02-15 14:58:07,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74243.38 MB 2025-02-15 14:58:07,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59557.02 MB 2025-02-15 14:58:07,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14686.36 MB 2025-02-15 14:58:07,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57624.75 MB 2025-02-15 14:58:07,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 14:58:07,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 14:58:07,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 14:58:07,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:07,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48744.77 MB 2025-02-15 14:58:07,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41833.82 MB 2025-02-15 14:58:07,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6910.96 MB 2025-02-15 14:58:07,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59557.02 MB 2025-02-15 14:58:07,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71963.77 MB 2025-02-15 14:58:07,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12406.75 MB 2025-02-15 14:58:07,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67398.70 MB 2025-02-15 14:58:09,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 14:58:09,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 14:58:09,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 14:58:09,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:09,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41833.82 MB 2025-02-15 14:58:09,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42364.66 MB 2025-02-15 14:58:09,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 14:58:09,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71963.77 MB 2025-02-15 14:58:09,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53076.82 MB 2025-02-15 14:58:09,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18886.95 MB 2025-02-15 14:58:09,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46343.20 MB 2025-02-15 14:58:09,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 14:58:09,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 14:58:09,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:58:09,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:09,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42364.66 MB 2025-02-15 14:58:09,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44254.01 MB 2025-02-15 14:58:09,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-15 14:58:09,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53076.82 MB 2025-02-15 14:58:09,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53076.82 MB 2025-02-15 14:58:09,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:58:09,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45671.44 MB 2025-02-15 14:58:10,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 14:58:10,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 14:58:10,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 14:58:10,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44254.01 MB 2025-02-15 14:58:10,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46495.87 MB 2025-02-15 14:58:10,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 14:58:10,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53076.82 MB 2025-02-15 14:58:10,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54966.35 MB 2025-02-15 14:58:10,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 14:58:10,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52040.15 MB 2025-02-15 14:58:10,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 14:58:10,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 14:58:10,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 14:58:10,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42364.66 MB 2025-02-15 14:58:10,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46495.87 MB 2025-02-15 14:58:10,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-15 14:58:10,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53076.82 MB 2025-02-15 14:58:10,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54966.35 MB 2025-02-15 14:58:10,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 14:58:10,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52040.15 MB 2025-02-15 14:58:10,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 14:58:10,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 14:58:10,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 14:58:10,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47203.66 MB 2025-02-15 14:58:10,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47970.66 MB 2025-02-15 14:58:10,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 14:58:10,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54966.35 MB 2025-02-15 14:58:10,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55383.69 MB 2025-02-15 14:58:10,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 14:58:10,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48678.45 MB 2025-02-15 14:58:10,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 14:58:10,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 14:58:10,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 14:58:10,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48383.55 MB 2025-02-15 14:58:10,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48590.21 MB 2025-02-15 14:58:10,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.67 MB 2025-02-15 14:58:10,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55383.69 MB 2025-02-15 14:58:10,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55383.69 MB 2025-02-15 14:58:10,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:58:10,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48800.98 MB 2025-02-15 14:58:10,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 14:58:10,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 14:58:10,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.84 seconds 2025-02-15 14:58:10,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35885.22 MB 2025-02-15 14:58:10,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48790.87 MB 2025-02-15 14:58:10,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12905.64 MB 2025-02-15 14:58:10,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74243.38 MB 2025-02-15 14:58:10,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55383.69 MB 2025-02-15 14:58:10,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18859.69 MB 2025-02-15 14:58:10,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48800.98 MB 2025-02-15 14:58:10,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 14:58:10,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 14:58:10,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 14:58:10,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48790.87 MB 2025-02-15 14:58:10,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48891.13 MB 2025-02-15 14:58:10,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-15 14:58:10,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55383.69 MB 2025-02-15 14:58:10,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55383.69 MB 2025-02-15 14:58:10,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 14:58:10,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49492.67 MB 2025-02-15 14:58:10,603 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 14:58:10,603 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 14:58:10,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 14:58:10,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 14:58:10,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 14:58:10,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 14:58:10,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37147.73 MB 2025-02-15 14:58:10,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41333.49 MB 2025-02-15 14:58:10,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.76 MB 2025-02-15 14:58:10,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55383.69 MB 2025-02-15 14:58:10,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59569.60 MB 2025-02-15 14:58:10,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-15 14:58:10,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45519.41 MB 2025-02-15 14:58:10,767 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 14:58:10,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,768 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,769 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 14:58:10,774 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 14:58:10,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,775 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 14:58:10,775 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 14:58:10,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,776 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,776 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:58:10,782 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 14:58:10,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,783 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,783 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:58:10,783 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 14:58:10,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,784 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:58:10,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,784 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 14:58:10,784 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 14:58:10,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,785 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 14:58:10,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,788 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,789 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,790 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 14:58:10,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 14:58:10,803 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:14,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:14,946 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:14,954 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:00:14,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:14,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:00:14,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:14,958 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:00:48,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:00:48,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:00:48,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.50 seconds 2025-02-15 15:00:48,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:48,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44691.87 MB 2025-02-15 15:00:48,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52343.07 MB 2025-02-15 15:00:48,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7651.20 MB 2025-02-15 15:00:48,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75134.66 MB 2025-02-15 15:00:48,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65036.88 MB 2025-02-15 15:00:48,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10097.79 MB 2025-02-15 15:00:48,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61184.51 MB 2025-02-15 15:00:48,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:00:48,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:00:48,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 15:00:48,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:48,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52343.07 MB 2025-02-15 15:00:48,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43675.42 MB 2025-02-15 15:00:48,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8667.65 MB 2025-02-15 15:00:48,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65036.88 MB 2025-02-15 15:00:48,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79683.39 MB 2025-02-15 15:00:48,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14646.51 MB 2025-02-15 15:00:48,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74366.75 MB 2025-02-15 15:00:50,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:00:50,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:00:50,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:00:50,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43675.42 MB 2025-02-15 15:00:50,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44206.26 MB 2025-02-15 15:00:50,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:00:50,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79683.39 MB 2025-02-15 15:00:50,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53194.26 MB 2025-02-15 15:00:50,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26489.13 MB 2025-02-15 15:00:50,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48184.81 MB 2025-02-15 15:00:50,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:00:50,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:00:50,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:00:50,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44206.26 MB 2025-02-15 15:00:50,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46095.67 MB 2025-02-15 15:00:50,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.40 MB 2025-02-15 15:00:50,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53194.26 MB 2025-02-15 15:00:50,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53196.36 MB 2025-02-15 15:00:50,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 15:00:50,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47513.09 MB 2025-02-15 15:00:50,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:00:50,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:00:50,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:00:50,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46095.67 MB 2025-02-15 15:00:50,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48337.52 MB 2025-02-15 15:00:50,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:00:50,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53196.36 MB 2025-02-15 15:00:50,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56027.51 MB 2025-02-15 15:00:50,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 15:00:50,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53881.80 MB 2025-02-15 15:00:50,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:00:50,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:00:50,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:00:50,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44206.26 MB 2025-02-15 15:00:50,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48337.52 MB 2025-02-15 15:00:50,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.26 MB 2025-02-15 15:00:50,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53194.26 MB 2025-02-15 15:00:50,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56027.51 MB 2025-02-15 15:00:50,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 15:00:50,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53881.80 MB 2025-02-15 15:00:50,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:00:50,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:00:50,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:00:50,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49045.31 MB 2025-02-15 15:00:50,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49812.31 MB 2025-02-15 15:00:50,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:00:50,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56027.51 MB 2025-02-15 15:00:50,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56444.85 MB 2025-02-15 15:00:50,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:00:50,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50520.10 MB 2025-02-15 15:00:50,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:00:50,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:00:50,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:00:50,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50225.20 MB 2025-02-15 15:00:50,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50432.60 MB 2025-02-15 15:00:50,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.40 MB 2025-02-15 15:00:50,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56444.85 MB 2025-02-15 15:00:50,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56444.85 MB 2025-02-15 15:00:50,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:00:50,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50630.94 MB 2025-02-15 15:00:50,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:00:50,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:00:50,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.98 seconds 2025-02-15 15:00:50,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:50,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37159.29 MB 2025-02-15 15:00:50,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50633.68 MB 2025-02-15 15:00:50,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13474.38 MB 2025-02-15 15:00:50,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75134.66 MB 2025-02-15 15:00:50,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56444.85 MB 2025-02-15 15:00:50,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18689.82 MB 2025-02-15 15:00:50,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50633.68 MB 2025-02-15 15:00:51,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:00:51,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:00:51,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:00:51,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:51,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50633.68 MB 2025-02-15 15:00:51,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50734.14 MB 2025-02-15 15:00:51,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 15:00:51,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56444.85 MB 2025-02-15 15:00:51,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56444.85 MB 2025-02-15 15:00:51,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:00:51,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51336.94 MB 2025-02-15 15:00:51,219 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:00:51,219 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:00:51,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:00:51,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:00:51,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:00:51,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:00:51,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38422.21 MB 2025-02-15 15:00:51,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42616.70 MB 2025-02-15 15:00:51,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 15:00:51,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56444.85 MB 2025-02-15 15:00:51,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56444.85 MB 2025-02-15 15:00:51,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:00:51,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46810.67 MB 2025-02-15 15:00:51,386 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:00:51,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:00:51,393 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:00:51,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,394 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:00:51,394 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:00:51,395 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,395 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,396 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:51,401 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:00:51,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,402 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,402 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:51,402 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:00:51,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,403 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:51,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,403 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:00:51,403 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:00:51,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,404 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:00:51,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,408 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,410 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,411 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:00:51,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:00:51,463 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:02:29,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:02:29,844 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:02:29,849 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:02:29,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:02:29,850 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2879, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:02:29,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:02:29,851 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2879, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:03:14,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:03:14,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:03:14,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.32 seconds 2025-02-15 15:03:14,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:14,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49809.26 MB 2025-02-15 15:03:14,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 59997.88 MB 2025-02-15 15:03:14,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10188.62 MB 2025-02-15 15:03:14,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72131.54 MB 2025-02-15 15:03:14,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74885.10 MB 2025-02-15 15:03:14,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2753.56 MB 2025-02-15 15:03:14,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70186.50 MB 2025-02-15 15:03:14,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:03:14,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:03:14,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:03:14,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:14,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 59997.88 MB 2025-02-15 15:03:14,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47524.10 MB 2025-02-15 15:03:14,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12473.79 MB 2025-02-15 15:03:14,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74885.10 MB 2025-02-15 15:03:14,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 93562.34 MB 2025-02-15 15:03:14,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18677.24 MB 2025-02-15 15:03:14,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 88087.10 MB 2025-02-15 15:03:16,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:03:16,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:03:16,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 15:03:16,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47524.10 MB 2025-02-15 15:03:16,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48054.94 MB 2025-02-15 15:03:16,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:03:16,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 93562.34 MB 2025-02-15 15:03:16,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64695.04 MB 2025-02-15 15:03:16,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28867.30 MB 2025-02-15 15:03:16,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52033.49 MB 2025-02-15 15:03:16,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:03:16,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:03:16,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:03:16,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48054.94 MB 2025-02-15 15:03:16,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49944.39 MB 2025-02-15 15:03:16,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.45 MB 2025-02-15 15:03:16,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64695.04 MB 2025-02-15 15:03:16,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64695.04 MB 2025-02-15 15:03:16,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:16,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51361.82 MB 2025-02-15 15:03:16,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:03:16,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:03:16,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:03:16,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49944.39 MB 2025-02-15 15:03:16,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52186.25 MB 2025-02-15 15:03:16,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:03:16,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64695.04 MB 2025-02-15 15:03:16,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64695.04 MB 2025-02-15 15:03:16,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:16,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57730.53 MB 2025-02-15 15:03:16,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:03:16,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:03:16,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:03:16,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48054.94 MB 2025-02-15 15:03:16,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52186.25 MB 2025-02-15 15:03:16,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.31 MB 2025-02-15 15:03:16,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64695.04 MB 2025-02-15 15:03:16,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64695.04 MB 2025-02-15 15:03:16,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:16,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57730.53 MB 2025-02-15 15:03:16,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:03:16,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:03:16,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 15:03:16,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52894.03 MB 2025-02-15 15:03:16,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53661.04 MB 2025-02-15 15:03:16,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:03:16,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64695.04 MB 2025-02-15 15:03:16,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65112.38 MB 2025-02-15 15:03:16,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:03:16,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54368.82 MB 2025-02-15 15:03:16,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:03:16,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:03:16,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:03:16,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54073.92 MB 2025-02-15 15:03:16,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54281.17 MB 2025-02-15 15:03:16,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.24 MB 2025-02-15 15:03:16,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65112.38 MB 2025-02-15 15:03:16,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65112.38 MB 2025-02-15 15:03:16,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:16,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54490.06 MB 2025-02-15 15:03:16,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:03:16,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:03:16,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.04 seconds 2025-02-15 15:03:16,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:16,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39778.60 MB 2025-02-15 15:03:16,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54482.24 MB 2025-02-15 15:03:16,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14703.65 MB 2025-02-15 15:03:16,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72131.54 MB 2025-02-15 15:03:16,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65112.38 MB 2025-02-15 15:03:16,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7019.17 MB 2025-02-15 15:03:16,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54490.06 MB 2025-02-15 15:03:17,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:03:17,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:03:17,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:03:17,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:17,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54482.24 MB 2025-02-15 15:03:17,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54582.71 MB 2025-02-15 15:03:17,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 15:03:17,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65112.38 MB 2025-02-15 15:03:17,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65112.38 MB 2025-02-15 15:03:17,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:17,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55185.51 MB 2025-02-15 15:03:17,175 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:03:17,176 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:03:17,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:03:17,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:03:17,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:03:17,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:03:17,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41041.52 MB 2025-02-15 15:03:17,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45236.00 MB 2025-02-15 15:03:17,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 15:03:17,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65112.38 MB 2025-02-15 15:03:17,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65112.38 MB 2025-02-15 15:03:17,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:03:17,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49429.97 MB 2025-02-15 15:03:17,340 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:03:17,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:03:17,347 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:03:17,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,348 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:03:17,348 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:03:17,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,349 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,349 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:03:17,355 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:03:17,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,356 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,356 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:03:17,356 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:03:17,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,357 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:03:17,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,357 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:03:17,357 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:03:17,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,358 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:03:17,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,363 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,365 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,366 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:03:17,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:03:17,380 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:04:56,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:04:56,612 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:04:56,617 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:04:56,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:04:56,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2189, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:04:56,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:04:56,619 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2189, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:05:30,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:05:30,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:05:30,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.79 seconds 2025-02-15 15:05:30,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:30,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45123.84 MB 2025-02-15 15:05:30,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52870.59 MB 2025-02-15 15:05:30,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7746.75 MB 2025-02-15 15:05:30,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80920.71 MB 2025-02-15 15:05:30,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64816.68 MB 2025-02-15 15:05:30,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16104.03 MB 2025-02-15 15:05:30,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61842.97 MB 2025-02-15 15:05:30,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:05:30,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:05:30,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 15:05:30,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:30,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52870.59 MB 2025-02-15 15:05:30,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44059.61 MB 2025-02-15 15:05:30,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8810.98 MB 2025-02-15 15:05:30,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64816.68 MB 2025-02-15 15:05:30,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78506.89 MB 2025-02-15 15:05:30,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13690.21 MB 2025-02-15 15:05:30,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73457.84 MB 2025-02-15 15:05:32,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:05:32,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:05:32,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:05:32,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44059.61 MB 2025-02-15 15:05:32,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44590.45 MB 2025-02-15 15:05:32,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:05:32,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78506.89 MB 2025-02-15 15:05:32,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64816.68 MB 2025-02-15 15:05:32,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13690.21 MB 2025-02-15 15:05:32,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48569.00 MB 2025-02-15 15:05:32,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:05:32,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:05:32,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:05:32,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44590.45 MB 2025-02-15 15:05:32,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46479.94 MB 2025-02-15 15:05:32,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:05:32,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64816.68 MB 2025-02-15 15:05:32,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64816.68 MB 2025-02-15 15:05:32,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:32,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47897.37 MB 2025-02-15 15:05:32,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:05:32,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:05:32,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:05:32,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46479.94 MB 2025-02-15 15:05:32,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48721.80 MB 2025-02-15 15:05:32,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:05:32,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64816.68 MB 2025-02-15 15:05:32,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64816.68 MB 2025-02-15 15:05:32,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:32,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54266.08 MB 2025-02-15 15:05:32,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:05:32,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:05:32,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:05:32,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44590.45 MB 2025-02-15 15:05:32,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48721.80 MB 2025-02-15 15:05:32,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:05:32,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64816.68 MB 2025-02-15 15:05:32,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64816.68 MB 2025-02-15 15:05:32,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:32,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54266.08 MB 2025-02-15 15:05:32,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:05:32,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:05:32,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:05:32,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49429.59 MB 2025-02-15 15:05:32,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50196.59 MB 2025-02-15 15:05:32,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:05:32,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64816.68 MB 2025-02-15 15:05:32,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65234.01 MB 2025-02-15 15:05:32,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:05:32,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50904.38 MB 2025-02-15 15:05:32,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:05:32,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:05:32,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:05:32,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50609.48 MB 2025-02-15 15:05:32,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50815.79 MB 2025-02-15 15:05:32,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.32 MB 2025-02-15 15:05:32,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65234.01 MB 2025-02-15 15:05:32,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65234.01 MB 2025-02-15 15:05:32,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:32,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51017.73 MB 2025-02-15 15:05:32,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:05:32,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:05:32,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.26 seconds 2025-02-15 15:05:32,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:32,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37497.19 MB 2025-02-15 15:05:32,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51016.13 MB 2025-02-15 15:05:32,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13518.94 MB 2025-02-15 15:05:32,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80920.71 MB 2025-02-15 15:05:32,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65234.01 MB 2025-02-15 15:05:32,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15686.70 MB 2025-02-15 15:05:32,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51017.73 MB 2025-02-15 15:05:33,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:05:33,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:05:33,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:05:33,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:33,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51016.13 MB 2025-02-15 15:05:33,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51116.23 MB 2025-02-15 15:05:33,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.10 MB 2025-02-15 15:05:33,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65234.01 MB 2025-02-15 15:05:33,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65234.01 MB 2025-02-15 15:05:33,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:33,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51716.82 MB 2025-02-15 15:05:33,169 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 15:05:33,169 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:05:33,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:05:33,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:05:33,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:05:33,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:05:33,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38759.37 MB 2025-02-15 15:05:33,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42938.47 MB 2025-02-15 15:05:33,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.09 MB 2025-02-15 15:05:33,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65234.01 MB 2025-02-15 15:05:33,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65234.01 MB 2025-02-15 15:05:33,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:05:33,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47117.05 MB 2025-02-15 15:05:33,341 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 15:05:33,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,343 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,344 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:05:33,348 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:05:33,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,350 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:05:33,350 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:05:33,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,350 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,351 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:05:33,357 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:05:33,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,358 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,358 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:05:33,358 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:05:33,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,359 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:05:33,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,359 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:05:33,359 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:05:33,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,360 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:05:33,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,363 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,364 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,365 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:05:33,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:05:33,387 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:07:26,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:07:26,252 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:07:26,261 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:07:26,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:07:26,264 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2086, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:07:26,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:07:26,265 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2086, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:07:58,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:07:58,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:07:58,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.04 seconds 2025-02-15 15:07:58,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:07:58,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44527.36 MB 2025-02-15 15:07:58,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51909.59 MB 2025-02-15 15:07:58,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7382.24 MB 2025-02-15 15:07:58,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81161.88 MB 2025-02-15 15:07:58,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64938.31 MB 2025-02-15 15:07:58,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16223.57 MB 2025-02-15 15:07:58,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60793.50 MB 2025-02-15 15:07:58,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:07:58,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:07:58,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 15:07:58,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:07:58,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51909.59 MB 2025-02-15 15:07:58,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43645.38 MB 2025-02-15 15:07:58,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8264.21 MB 2025-02-15 15:07:58,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 15:07:58,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79035.37 MB 2025-02-15 15:07:58,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14097.06 MB 2025-02-15 15:07:58,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73109.75 MB 2025-02-15 15:08:00,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:08:00,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:08:00,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 15:08:00,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43645.38 MB 2025-02-15 15:08:00,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44176.22 MB 2025-02-15 15:08:00,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:08:00,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79035.37 MB 2025-02-15 15:08:00,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64938.31 MB 2025-02-15 15:08:00,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14097.06 MB 2025-02-15 15:08:00,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48154.77 MB 2025-02-15 15:08:00,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:08:00,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:08:00,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:08:00,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44176.22 MB 2025-02-15 15:08:00,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46065.72 MB 2025-02-15 15:08:00,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:08:00,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 15:08:00,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64938.31 MB 2025-02-15 15:08:00,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:00,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47483.14 MB 2025-02-15 15:08:00,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:08:00,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:08:00,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:08:00,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46065.72 MB 2025-02-15 15:08:00,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48307.57 MB 2025-02-15 15:08:00,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:08:00,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 15:08:00,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64938.31 MB 2025-02-15 15:08:00,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:00,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53851.85 MB 2025-02-15 15:08:00,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:08:00,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:08:00,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:08:00,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44176.22 MB 2025-02-15 15:08:00,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48307.57 MB 2025-02-15 15:08:00,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:08:00,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 15:08:00,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64938.31 MB 2025-02-15 15:08:00,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:00,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53851.85 MB 2025-02-15 15:08:00,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:08:00,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:08:00,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:08:00,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49015.36 MB 2025-02-15 15:08:00,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49782.36 MB 2025-02-15 15:08:00,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:08:00,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64938.31 MB 2025-02-15 15:08:00,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65353.55 MB 2025-02-15 15:08:00,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:08:00,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50490.15 MB 2025-02-15 15:08:00,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:08:00,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:08:00,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:08:00,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50195.25 MB 2025-02-15 15:08:00,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50401.95 MB 2025-02-15 15:08:00,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.69 MB 2025-02-15 15:08:00,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65353.55 MB 2025-02-15 15:08:00,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65353.55 MB 2025-02-15 15:08:00,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:00,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50609.50 MB 2025-02-15 15:08:00,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:08:00,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:08:00,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.53 seconds 2025-02-15 15:08:00,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:00,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37259.56 MB 2025-02-15 15:08:00,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50602.63 MB 2025-02-15 15:08:00,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13343.06 MB 2025-02-15 15:08:00,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81161.88 MB 2025-02-15 15:08:00,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65353.55 MB 2025-02-15 15:08:00,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15808.33 MB 2025-02-15 15:08:00,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50609.50 MB 2025-02-15 15:08:01,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:08:01,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:08:01,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:08:01,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:01,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50602.63 MB 2025-02-15 15:08:01,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50702.90 MB 2025-02-15 15:08:01,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.27 MB 2025-02-15 15:08:01,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65353.55 MB 2025-02-15 15:08:01,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65353.55 MB 2025-02-15 15:08:01,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:01,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51304.52 MB 2025-02-15 15:08:01,084 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 15:08:01,084 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:08:01,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:08:01,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:08:01,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:08:01,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:08:01,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38522.09 MB 2025-02-15 15:08:01,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42708.37 MB 2025-02-15 15:08:01,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.28 MB 2025-02-15 15:08:01,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65353.55 MB 2025-02-15 15:08:01,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65353.55 MB 2025-02-15 15:08:01,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:08:01,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46894.13 MB 2025-02-15 15:08:01,248 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 15:08:01,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,250 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,251 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:08:01,255 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:08:01,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,256 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:08:01,257 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:08:01,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,257 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,258 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:08:01,264 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:08:01,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,264 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,265 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:08:01,265 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:08:01,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,265 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:08:01,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,266 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:08:01,266 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:08:01,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,266 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:08:01,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,270 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,271 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,272 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:08:01,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:08:01,286 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:01,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:01,610 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:01,615 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:10:01,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:01,617 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:10:01,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:01,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:10:43,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:10:43,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:10:43,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.17 seconds 2025-02-15 15:10:43,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:43,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49220.02 MB 2025-02-15 15:10:43,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 58923.80 MB 2025-02-15 15:10:43,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9703.78 MB 2025-02-15 15:10:43,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81403.05 MB 2025-02-15 15:10:43,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65059.95 MB 2025-02-15 15:10:43,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16343.11 MB 2025-02-15 15:10:43,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68627.59 MB 2025-02-15 15:10:44,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:10:44,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:10:44,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:10:44,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:44,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 58923.80 MB 2025-02-15 15:10:44,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47178.32 MB 2025-02-15 15:10:44,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11745.48 MB 2025-02-15 15:10:44,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65059.95 MB 2025-02-15 15:10:44,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 86408.95 MB 2025-02-15 15:10:44,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21349.01 MB 2025-02-15 15:10:44,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 87833.94 MB 2025-02-15 15:10:46,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:10:46,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:10:46,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:10:46,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47178.32 MB 2025-02-15 15:10:46,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47709.16 MB 2025-02-15 15:10:46,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:10:46,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 86408.95 MB 2025-02-15 15:10:46,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67184.36 MB 2025-02-15 15:10:46,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19224.59 MB 2025-02-15 15:10:46,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51687.71 MB 2025-02-15 15:10:46,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:10:46,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:10:46,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:10:46,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47709.16 MB 2025-02-15 15:10:46,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49598.65 MB 2025-02-15 15:10:46,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:10:46,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67184.36 MB 2025-02-15 15:10:46,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67184.36 MB 2025-02-15 15:10:46,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51016.08 MB 2025-02-15 15:10:46,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:10:46,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:10:46,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:10:46,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49598.65 MB 2025-02-15 15:10:46,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51840.51 MB 2025-02-15 15:10:46,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:10:46,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67184.36 MB 2025-02-15 15:10:46,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67184.36 MB 2025-02-15 15:10:46,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57384.79 MB 2025-02-15 15:10:46,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:10:46,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:10:46,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:10:46,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47709.16 MB 2025-02-15 15:10:46,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51840.51 MB 2025-02-15 15:10:46,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:10:46,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67184.36 MB 2025-02-15 15:10:46,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67184.36 MB 2025-02-15 15:10:46,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57384.79 MB 2025-02-15 15:10:46,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:10:46,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:10:46,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:10:46,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52548.30 MB 2025-02-15 15:10:46,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53315.30 MB 2025-02-15 15:10:46,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:10:46,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67184.36 MB 2025-02-15 15:10:46,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67599.60 MB 2025-02-15 15:10:46,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:10:46,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54023.09 MB 2025-02-15 15:10:46,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:10:46,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:10:46,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:10:46,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53728.19 MB 2025-02-15 15:10:46,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53934.79 MB 2025-02-15 15:10:46,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.60 MB 2025-02-15 15:10:46,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67599.60 MB 2025-02-15 15:10:46,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67599.60 MB 2025-02-15 15:10:46,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54165.38 MB 2025-02-15 15:10:46,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:10:46,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:10:46,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.80 seconds 2025-02-15 15:10:46,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39666.67 MB 2025-02-15 15:10:46,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54135.37 MB 2025-02-15 15:10:46,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14468.70 MB 2025-02-15 15:10:46,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81403.05 MB 2025-02-15 15:10:46,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67599.60 MB 2025-02-15 15:10:46,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13803.45 MB 2025-02-15 15:10:46,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54165.38 MB 2025-02-15 15:10:46,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:10:46,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:10:46,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:10:46,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54135.37 MB 2025-02-15 15:10:46,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54235.59 MB 2025-02-15 15:10:46,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-15 15:10:46,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67599.60 MB 2025-02-15 15:10:46,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67599.60 MB 2025-02-15 15:10:46,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54836.91 MB 2025-02-15 15:10:46,701 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 15:10:46,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:10:46,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:10:46,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:10:46,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:10:46,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:10:46,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40929.10 MB 2025-02-15 15:10:46,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45113.33 MB 2025-02-15 15:10:46,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-15 15:10:46,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67599.60 MB 2025-02-15 15:10:46,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67599.60 MB 2025-02-15 15:10:46,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:10:46,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49297.04 MB 2025-02-15 15:10:46,865 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 15:10:46,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,866 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:10:46,872 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:10:46,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,873 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:10:46,873 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:10:46,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,873 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,874 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:46,880 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:10:46,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,880 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,881 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:46,881 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:10:46,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,881 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:46,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,882 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:10:46,882 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:10:46,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,882 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:10:46,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,886 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,887 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,887 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:10:46,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:10:46,908 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:12:23,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:12:23,287 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:12:23,292 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:12:23,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:12:23,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2786, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:12:23,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:12:23,294 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2786, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:13:06,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:13:06,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:13:06,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.23 seconds 2025-02-15 15:13:06,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:06,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49648.47 MB 2025-02-15 15:13:06,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 59507.97 MB 2025-02-15 15:13:06,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9859.50 MB 2025-02-15 15:13:06,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 83770.74 MB 2025-02-15 15:13:06,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75042.39 MB 2025-02-15 15:13:06,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8728.35 MB 2025-02-15 15:13:06,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69367.46 MB 2025-02-15 15:13:06,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:13:06,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:13:06,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:13:06,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:06,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 59507.97 MB 2025-02-15 15:13:06,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47527.86 MB 2025-02-15 15:13:06,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11980.10 MB 2025-02-15 15:13:06,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75042.39 MB 2025-02-15 15:13:06,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 94527.03 MB 2025-02-15 15:13:06,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19484.64 MB 2025-02-15 15:13:06,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 88808.64 MB 2025-02-15 15:13:08,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:13:08,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:13:08,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:13:08,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:08,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47527.86 MB 2025-02-15 15:13:08,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48058.70 MB 2025-02-15 15:13:08,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:13:08,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94527.03 MB 2025-02-15 15:13:08,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65181.58 MB 2025-02-15 15:13:08,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29345.45 MB 2025-02-15 15:13:08,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52037.25 MB 2025-02-15 15:13:08,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:13:08,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:13:08,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:13:08,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:08,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48058.70 MB 2025-02-15 15:13:08,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49948.20 MB 2025-02-15 15:13:08,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:13:08,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65181.58 MB 2025-02-15 15:13:08,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65181.58 MB 2025-02-15 15:13:08,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:08,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51365.63 MB 2025-02-15 15:13:08,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:13:08,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:13:08,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:13:08,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:08,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49948.20 MB 2025-02-15 15:13:08,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52190.05 MB 2025-02-15 15:13:08,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:13:08,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65181.58 MB 2025-02-15 15:13:08,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65181.58 MB 2025-02-15 15:13:08,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:08,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57734.33 MB 2025-02-15 15:13:08,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:13:08,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:13:08,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:13:08,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:08,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48058.70 MB 2025-02-15 15:13:08,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52190.05 MB 2025-02-15 15:13:08,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:13:08,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65181.58 MB 2025-02-15 15:13:08,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65181.58 MB 2025-02-15 15:13:08,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:08,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57734.33 MB 2025-02-15 15:13:09,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:13:09,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:13:09,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:13:09,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:09,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52897.84 MB 2025-02-15 15:13:09,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53664.84 MB 2025-02-15 15:13:09,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:13:09,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65181.58 MB 2025-02-15 15:13:09,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65596.82 MB 2025-02-15 15:13:09,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:13:09,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54372.63 MB 2025-02-15 15:13:09,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:13:09,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:13:09,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:13:09,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:09,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54077.73 MB 2025-02-15 15:13:09,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54283.51 MB 2025-02-15 15:13:09,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.77 MB 2025-02-15 15:13:09,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65596.82 MB 2025-02-15 15:13:09,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65596.82 MB 2025-02-15 15:13:09,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:09,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54515.14 MB 2025-02-15 15:13:09,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:13:09,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:13:09,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.78 seconds 2025-02-15 15:13:09,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:09,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39941.82 MB 2025-02-15 15:13:09,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54483.52 MB 2025-02-15 15:13:09,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14541.70 MB 2025-02-15 15:13:09,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 83770.74 MB 2025-02-15 15:13:09,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65596.82 MB 2025-02-15 15:13:09,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18173.92 MB 2025-02-15 15:13:09,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54515.14 MB 2025-02-15 15:13:09,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:13:09,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:13:09,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:13:09,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:09,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54483.52 MB 2025-02-15 15:13:09,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54583.46 MB 2025-02-15 15:13:09,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.94 MB 2025-02-15 15:13:09,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65596.82 MB 2025-02-15 15:13:09,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65596.82 MB 2025-02-15 15:13:09,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:09,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55183.09 MB 2025-02-15 15:13:09,363 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-15 15:13:09,363 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:13:09,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:13:09,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:13:09,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:13:09,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:13:09,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41203.68 MB 2025-02-15 15:13:09,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45376.11 MB 2025-02-15 15:13:09,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4172.43 MB 2025-02-15 15:13:09,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65596.82 MB 2025-02-15 15:13:09,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65596.82 MB 2025-02-15 15:13:09,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:13:09,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49548.02 MB 2025-02-15 15:13:09,533 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-15 15:13:09,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,534 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,535 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:13:09,540 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:13:09,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,541 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:13:09,541 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:13:09,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,542 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,542 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:13:09,548 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:13:09,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,549 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,549 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:13:09,549 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:13:09,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,550 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:13:09,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,550 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:13:09,550 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:13:09,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,551 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:13:09,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,554 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,555 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,557 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:13:09,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:13:09,647 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:14,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:14,613 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:14,618 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:15:14,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:14,619 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:15:14,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:14,620 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:15:48,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:15:48,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:15:48,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.52 seconds 2025-02-15 15:15:48,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:48,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45603.21 MB 2025-02-15 15:15:48,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53346.42 MB 2025-02-15 15:15:48,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7743.21 MB 2025-02-15 15:15:48,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81889.59 MB 2025-02-15 15:15:48,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65303.22 MB 2025-02-15 15:15:48,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16586.38 MB 2025-02-15 15:15:48,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62322.34 MB 2025-02-15 15:15:48,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:15:48,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:15:48,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 15:15:48,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:48,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53346.42 MB 2025-02-15 15:15:48,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44540.75 MB 2025-02-15 15:15:48,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8805.67 MB 2025-02-15 15:15:48,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65303.22 MB 2025-02-15 15:15:48,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78628.52 MB 2025-02-15 15:15:48,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13325.30 MB 2025-02-15 15:15:48,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73385.78 MB 2025-02-15 15:15:50,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:15:50,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:15:50,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:15:50,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44540.75 MB 2025-02-15 15:15:50,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45071.59 MB 2025-02-15 15:15:50,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:15:50,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78628.52 MB 2025-02-15 15:15:50,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65303.22 MB 2025-02-15 15:15:50,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13325.30 MB 2025-02-15 15:15:50,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49050.14 MB 2025-02-15 15:15:50,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:15:50,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:15:50,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:15:50,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45071.59 MB 2025-02-15 15:15:50,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46961.08 MB 2025-02-15 15:15:50,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:15:50,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65303.22 MB 2025-02-15 15:15:50,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65303.22 MB 2025-02-15 15:15:50,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48378.51 MB 2025-02-15 15:15:50,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:15:50,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:15:50,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:15:50,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46961.08 MB 2025-02-15 15:15:50,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49202.94 MB 2025-02-15 15:15:50,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:15:50,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65303.22 MB 2025-02-15 15:15:50,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65303.22 MB 2025-02-15 15:15:50,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54747.22 MB 2025-02-15 15:15:50,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:15:50,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:15:50,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:15:50,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45071.59 MB 2025-02-15 15:15:50,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49202.94 MB 2025-02-15 15:15:50,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:15:50,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65303.22 MB 2025-02-15 15:15:50,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65303.22 MB 2025-02-15 15:15:50,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54747.22 MB 2025-02-15 15:15:50,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:15:50,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:15:50,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:15:50,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49910.73 MB 2025-02-15 15:15:50,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50677.73 MB 2025-02-15 15:15:50,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:15:50,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65303.22 MB 2025-02-15 15:15:50,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65718.45 MB 2025-02-15 15:15:50,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:15:50,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51385.52 MB 2025-02-15 15:15:50,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:15:50,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:15:50,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:15:50,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51090.62 MB 2025-02-15 15:15:50,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51297.57 MB 2025-02-15 15:15:50,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.95 MB 2025-02-15 15:15:50,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65718.45 MB 2025-02-15 15:15:50,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65718.45 MB 2025-02-15 15:15:50,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51504.49 MB 2025-02-15 15:15:50,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:15:50,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:15:50,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.98 seconds 2025-02-15 15:15:50,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37980.04 MB 2025-02-15 15:15:50,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51498.23 MB 2025-02-15 15:15:50,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13518.18 MB 2025-02-15 15:15:50,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81889.59 MB 2025-02-15 15:15:50,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65718.45 MB 2025-02-15 15:15:50,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16171.14 MB 2025-02-15 15:15:50,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51504.49 MB 2025-02-15 15:15:50,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:15:50,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:15:50,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:15:50,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51498.23 MB 2025-02-15 15:15:50,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51598.48 MB 2025-02-15 15:15:50,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-15 15:15:50,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65718.45 MB 2025-02-15 15:15:50,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65718.45 MB 2025-02-15 15:15:50,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52200.03 MB 2025-02-15 15:15:50,882 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-15 15:15:50,882 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:15:50,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:15:50,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:15:50,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:15:50,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:15:50,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39242.54 MB 2025-02-15 15:15:50,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43428.31 MB 2025-02-15 15:15:50,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.76 MB 2025-02-15 15:15:50,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65718.45 MB 2025-02-15 15:15:50,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65718.45 MB 2025-02-15 15:15:50,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:15:50,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47613.56 MB 2025-02-15 15:15:51,045 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-15 15:15:51,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,047 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,048 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:15:51,052 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:15:51,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,053 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:15:51,054 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:15:51,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,054 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,055 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:51,061 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:15:51,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,061 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,062 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:51,062 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:15:51,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,062 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:51,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,063 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:15:51,063 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:15:51,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,063 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:15:51,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,067 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,068 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,069 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:15:51,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:15:51,092 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:09,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:09,172 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:09,177 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:17:09,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:09,178 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2789, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:17:09,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:09,179 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2789, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:17:52,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:17:52,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:17:52,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.35 seconds 2025-02-15 15:17:52,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:52,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49912.72 MB 2025-02-15 15:17:52,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 59782.83 MB 2025-02-15 15:17:52,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9870.11 MB 2025-02-15 15:17:52,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82132.86 MB 2025-02-15 15:17:52,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75294.05 MB 2025-02-15 15:17:52,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6838.81 MB 2025-02-15 15:17:52,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69652.95 MB 2025-02-15 15:17:52,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:17:52,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:17:52,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 15:17:52,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:52,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 59782.83 MB 2025-02-15 15:17:52,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47786.81 MB 2025-02-15 15:17:52,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11996.03 MB 2025-02-15 15:17:52,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75294.05 MB 2025-02-15 15:17:52,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 94036.30 MB 2025-02-15 15:17:52,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18742.25 MB 2025-02-15 15:17:52,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 87963.72 MB 2025-02-15 15:17:54,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:17:54,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:17:54,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:17:54,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:54,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47786.81 MB 2025-02-15 15:17:54,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48317.65 MB 2025-02-15 15:17:54,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:17:54,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94036.30 MB 2025-02-15 15:17:54,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65422.75 MB 2025-02-15 15:17:54,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28613.54 MB 2025-02-15 15:17:54,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52296.20 MB 2025-02-15 15:17:54,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:17:54,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:17:54,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:17:54,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:54,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48317.65 MB 2025-02-15 15:17:54,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50207.14 MB 2025-02-15 15:17:54,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:17:54,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65422.75 MB 2025-02-15 15:17:54,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65422.75 MB 2025-02-15 15:17:54,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:54,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51624.57 MB 2025-02-15 15:17:54,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:17:54,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:17:54,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:17:54,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:54,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50207.14 MB 2025-02-15 15:17:54,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52449.00 MB 2025-02-15 15:17:54,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:17:54,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65422.75 MB 2025-02-15 15:17:54,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65422.75 MB 2025-02-15 15:17:54,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:54,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57993.28 MB 2025-02-15 15:17:54,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:17:54,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:17:54,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:17:54,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:54,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48317.65 MB 2025-02-15 15:17:54,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52449.00 MB 2025-02-15 15:17:54,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:17:54,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65422.75 MB 2025-02-15 15:17:54,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65422.75 MB 2025-02-15 15:17:54,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:54,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57993.28 MB 2025-02-15 15:17:55,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:17:55,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:17:55,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:17:55,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:55,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53156.79 MB 2025-02-15 15:17:55,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53923.79 MB 2025-02-15 15:17:55,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:17:55,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65422.75 MB 2025-02-15 15:17:55,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65840.09 MB 2025-02-15 15:17:55,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:17:55,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54631.58 MB 2025-02-15 15:17:55,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:17:55,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:17:55,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:17:55,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:55,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54336.68 MB 2025-02-15 15:17:55,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54543.74 MB 2025-02-15 15:17:55,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.06 MB 2025-02-15 15:17:55,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65840.09 MB 2025-02-15 15:17:55,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65840.09 MB 2025-02-15 15:17:55,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:55,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54752.25 MB 2025-02-15 15:17:55,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:17:55,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:17:55,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.88 seconds 2025-02-15 15:17:55,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:55,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40195.62 MB 2025-02-15 15:17:55,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54744.81 MB 2025-02-15 15:17:55,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14549.19 MB 2025-02-15 15:17:55,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82132.86 MB 2025-02-15 15:17:55,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65840.09 MB 2025-02-15 15:17:55,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16292.77 MB 2025-02-15 15:17:55,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54752.25 MB 2025-02-15 15:17:55,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:17:55,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:17:55,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:17:55,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:55,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54744.81 MB 2025-02-15 15:17:55,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54845.28 MB 2025-02-15 15:17:55,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-15 15:17:55,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65840.09 MB 2025-02-15 15:17:55,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65840.09 MB 2025-02-15 15:17:55,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:55,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55448.08 MB 2025-02-15 15:17:55,339 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:17:55,340 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:17:55,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:17:55,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:17:55,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:17:55,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:17:55,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41458.54 MB 2025-02-15 15:17:55,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45653.03 MB 2025-02-15 15:17:55,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-15 15:17:55,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65840.09 MB 2025-02-15 15:17:55,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65840.09 MB 2025-02-15 15:17:55,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:17:55,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49847.00 MB 2025-02-15 15:17:55,505 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:17:55,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,506 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:17:55,511 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:17:55,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,512 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:17:55,513 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:17:55,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,513 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,514 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:55,520 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:17:55,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,520 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,521 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:55,521 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:17:55,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,521 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:55,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,522 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:17:55,522 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:17:55,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,522 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:17:55,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,525 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,526 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,527 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:17:55,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:17:55,548 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:09,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:09,525 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:09,530 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:20:09,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:09,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:20:09,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:09,532 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:20:45,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:20:45,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:20:45,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.38 seconds 2025-02-15 15:20:45,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:45,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47115.66 MB 2025-02-15 15:20:45,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55502.96 MB 2025-02-15 15:20:45,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8387.30 MB 2025-02-15 15:20:45,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82376.13 MB 2025-02-15 15:20:45,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65544.39 MB 2025-02-15 15:20:45,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16831.74 MB 2025-02-15 15:20:45,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64514.27 MB 2025-02-15 15:20:46,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:20:46,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:20:46,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 15:20:46,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:46,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 55502.96 MB 2025-02-15 15:20:46,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45730.32 MB 2025-02-15 15:20:46,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9772.64 MB 2025-02-15 15:20:46,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65544.39 MB 2025-02-15 15:20:46,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70485.28 MB 2025-02-15 15:20:46,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4940.89 MB 2025-02-15 15:20:46,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66330.37 MB 2025-02-15 15:20:47,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:20:47,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:20:47,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:20:47,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:47,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45730.32 MB 2025-02-15 15:20:47,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46261.16 MB 2025-02-15 15:20:47,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:20:47,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70485.28 MB 2025-02-15 15:20:47,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65544.39 MB 2025-02-15 15:20:47,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4940.89 MB 2025-02-15 15:20:47,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50239.71 MB 2025-02-15 15:20:47,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:20:47,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:20:47,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:20:47,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:47,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46261.16 MB 2025-02-15 15:20:47,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48150.66 MB 2025-02-15 15:20:47,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:20:47,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65544.39 MB 2025-02-15 15:20:47,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65544.39 MB 2025-02-15 15:20:47,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:47,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49568.09 MB 2025-02-15 15:20:48,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:20:48,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:20:48,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:20:48,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48150.66 MB 2025-02-15 15:20:48,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50392.51 MB 2025-02-15 15:20:48,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:20:48,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65544.39 MB 2025-02-15 15:20:48,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65544.39 MB 2025-02-15 15:20:48,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:48,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55936.79 MB 2025-02-15 15:20:48,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:20:48,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:20:48,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:20:48,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46261.16 MB 2025-02-15 15:20:48,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50392.51 MB 2025-02-15 15:20:48,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:20:48,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65544.39 MB 2025-02-15 15:20:48,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65544.39 MB 2025-02-15 15:20:48,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:48,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55936.79 MB 2025-02-15 15:20:48,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:20:48,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:20:48,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 15:20:48,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51100.30 MB 2025-02-15 15:20:48,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51867.30 MB 2025-02-15 15:20:48,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:20:48,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65544.39 MB 2025-02-15 15:20:48,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65961.72 MB 2025-02-15 15:20:48,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:20:48,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52575.09 MB 2025-02-15 15:20:48,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:20:48,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:20:48,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:20:48,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52280.19 MB 2025-02-15 15:20:48,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52486.32 MB 2025-02-15 15:20:48,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.13 MB 2025-02-15 15:20:48,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65961.72 MB 2025-02-15 15:20:48,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65961.72 MB 2025-02-15 15:20:48,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:48,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.83 MB 2025-02-15 15:20:48,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:20:48,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:20:48,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.95 seconds 2025-02-15 15:20:48,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38857.56 MB 2025-02-15 15:20:48,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52686.78 MB 2025-02-15 15:20:48,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13829.22 MB 2025-02-15 15:20:48,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82376.13 MB 2025-02-15 15:20:48,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65961.72 MB 2025-02-15 15:20:48,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16414.41 MB 2025-02-15 15:20:48,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.83 MB 2025-02-15 15:20:48,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:20:48,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:20:48,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:20:48,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52686.78 MB 2025-02-15 15:20:48,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52786.94 MB 2025-02-15 15:20:48,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.16 MB 2025-02-15 15:20:48,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65961.72 MB 2025-02-15 15:20:48,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65961.72 MB 2025-02-15 15:20:48,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:48,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53387.90 MB 2025-02-15 15:20:48,769 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 15:20:48,769 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:20:48,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:20:48,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:20:48,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:20:48,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:20:48,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40119.86 MB 2025-02-15 15:20:48,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44301.52 MB 2025-02-15 15:20:48,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.66 MB 2025-02-15 15:20:48,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65961.72 MB 2025-02-15 15:20:48,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65961.72 MB 2025-02-15 15:20:48,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:20:48,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48482.67 MB 2025-02-15 15:20:48,936 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 15:20:48,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,937 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,938 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:20:48,943 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:20:48,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,944 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:20:48,944 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:20:48,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,945 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,945 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:48,951 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:20:48,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,952 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,952 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:48,952 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:20:48,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,953 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:48,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,953 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:20:48,953 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:20:48,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,954 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:48,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,959 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,960 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,962 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:20:48,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:48,987 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:49,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:49,168 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:20:49,173 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:20:49,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:49,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2865, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:20:49,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:20:49,175 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2865, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:21:34,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:21:34,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:21:34,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.86 seconds 2025-02-15 15:21:34,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:34,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50685.77 MB 2025-02-15 15:21:34,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 60825.50 MB 2025-02-15 15:21:34,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10139.73 MB 2025-02-15 15:21:34,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82619.40 MB 2025-02-15 15:21:34,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75807.85 MB 2025-02-15 15:21:34,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6811.55 MB 2025-02-15 15:21:34,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70964.57 MB 2025-02-15 15:21:34,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:21:34,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:21:34,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 15:21:34,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:34,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 60825.50 MB 2025-02-15 15:21:34,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48425.44 MB 2025-02-15 15:21:34,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12400.06 MB 2025-02-15 15:21:34,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75807.85 MB 2025-02-15 15:21:34,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 94575.26 MB 2025-02-15 15:21:34,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18767.41 MB 2025-02-15 15:21:34,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 89049.46 MB 2025-02-15 15:21:36,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:21:36,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:21:36,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:21:36,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48425.44 MB 2025-02-15 15:21:36,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48956.28 MB 2025-02-15 15:21:36,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:21:36,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94575.26 MB 2025-02-15 15:21:36,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65668.12 MB 2025-02-15 15:21:36,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28907.14 MB 2025-02-15 15:21:36,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52934.83 MB 2025-02-15 15:21:36,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:21:36,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:21:36,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:21:36,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48956.28 MB 2025-02-15 15:21:36,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50845.77 MB 2025-02-15 15:21:36,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.49 MB 2025-02-15 15:21:36,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65668.12 MB 2025-02-15 15:21:36,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65668.12 MB 2025-02-15 15:21:36,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52263.20 MB 2025-02-15 15:21:36,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:21:36,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:21:36,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:21:36,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50845.77 MB 2025-02-15 15:21:36,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53087.63 MB 2025-02-15 15:21:36,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:21:36,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65668.12 MB 2025-02-15 15:21:36,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65668.12 MB 2025-02-15 15:21:36,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58631.91 MB 2025-02-15 15:21:36,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:21:36,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:21:36,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:21:36,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48956.28 MB 2025-02-15 15:21:36,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53087.63 MB 2025-02-15 15:21:36,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.35 MB 2025-02-15 15:21:36,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65668.12 MB 2025-02-15 15:21:36,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65668.12 MB 2025-02-15 15:21:36,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58631.91 MB 2025-02-15 15:21:36,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:21:36,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:21:36,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:21:36,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53795.42 MB 2025-02-15 15:21:36,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54562.42 MB 2025-02-15 15:21:36,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:21:36,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65668.12 MB 2025-02-15 15:21:36,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66083.36 MB 2025-02-15 15:21:36,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:21:36,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55270.21 MB 2025-02-15 15:21:36,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:21:36,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:21:36,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:21:36,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54975.31 MB 2025-02-15 15:21:36,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55181.91 MB 2025-02-15 15:21:36,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.60 MB 2025-02-15 15:21:36,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66083.36 MB 2025-02-15 15:21:36,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66083.36 MB 2025-02-15 15:21:36,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55390.62 MB 2025-02-15 15:21:36,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:21:36,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:21:36,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.43 seconds 2025-02-15 15:21:36,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40703.88 MB 2025-02-15 15:21:36,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55382.83 MB 2025-02-15 15:21:36,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14678.95 MB 2025-02-15 15:21:36,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82619.40 MB 2025-02-15 15:21:36,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66083.36 MB 2025-02-15 15:21:36,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16536.04 MB 2025-02-15 15:21:36,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55390.62 MB 2025-02-15 15:21:36,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:21:36,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:21:36,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:21:36,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 55382.83 MB 2025-02-15 15:21:36,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55483.22 MB 2025-02-15 15:21:36,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.39 MB 2025-02-15 15:21:36,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66083.36 MB 2025-02-15 15:21:36,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66083.36 MB 2025-02-15 15:21:36,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56085.58 MB 2025-02-15 15:21:36,898 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-15 15:21:36,899 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:21:36,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:21:36,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:21:36,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:21:36,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:21:36,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41966.72 MB 2025-02-15 15:21:36,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46158.12 MB 2025-02-15 15:21:36,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4191.41 MB 2025-02-15 15:21:36,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66083.36 MB 2025-02-15 15:21:36,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66083.36 MB 2025-02-15 15:21:36,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:21:36,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50349.02 MB 2025-02-15 15:21:37,071 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-15 15:21:37,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,072 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,073 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:21:37,078 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:21:37,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,079 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:21:37,079 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:21:37,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,081 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,081 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:21:37,087 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-15 15:21:37,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,088 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,089 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:21:37,089 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-15 15:21:37,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,089 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:21:37,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,090 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-15 15:21:37,090 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-15 15:21:37,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,090 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:21:37,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,097 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,099 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,101 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:21:37,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:37,127 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:21:44,606 - finetune_llama.py:467 - compute_metrics - INFO - In compute_metrics() 2025-02-15 15:21:44,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,607 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[0]: [(8192,), int64, CPU] 2025-02-15 15:21:44,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,608 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[1]: [(8192,), int64, CPU] 2025-02-15 15:21:44,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,608 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[2]: [(8192,), int64, CPU] 2025-02-15 15:21:44,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,609 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[3]: [(8192,), int64, CPU] 2025-02-15 15:21:44,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,610 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[4]: [(8192,), int64, CPU] 2025-02-15 15:21:44,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,610 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[5]: [(8192,), int64, CPU] 2025-02-15 15:21:44,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,611 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[6]: [(8192,), int64, CPU] 2025-02-15 15:21:44,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,611 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[7]: [(8192,), int64, CPU] 2025-02-15 15:21:44,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,612 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[8]: [(8192,), int64, CPU] 2025-02-15 15:21:44,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,612 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[9]: [(8192,), int64, CPU] 2025-02-15 15:21:44,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,613 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[10]: [(8192,), int64, CPU] 2025-02-15 15:21:44,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,614 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[11]: [(8192,), int64, CPU] 2025-02-15 15:21:44,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,615 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[12]: [(8192,), int64, CPU] 2025-02-15 15:21:44,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,615 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[13]: [(8192,), int64, CPU] 2025-02-15 15:21:44,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,616 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[14]: [(8192,), int64, CPU] 2025-02-15 15:21:44,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,616 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[15]: [(8192,), int64, CPU] 2025-02-15 15:21:44,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,617 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[16]: [(8192,), int64, CPU] 2025-02-15 15:21:44,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,617 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[17]: [(8192,), int64, CPU] 2025-02-15 15:21:44,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,618 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[18]: [(8192,), int64, CPU] 2025-02-15 15:21:44,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,618 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[19]: [(8192,), int64, CPU] 2025-02-15 15:21:44,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,619 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[20]: [(8192,), int64, CPU] 2025-02-15 15:21:44,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,619 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[21]: [(8192,), int64, CPU] 2025-02-15 15:21:44,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,620 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[22]: [(8192,), int64, CPU] 2025-02-15 15:21:44,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,620 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[23]: [(8192,), int64, CPU] 2025-02-15 15:21:44,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,621 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[24]: [(8192,), int64, CPU] 2025-02-15 15:21:44,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,621 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[25]: [(8192,), int64, CPU] 2025-02-15 15:21:44,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,622 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[26]: [(8192,), int64, CPU] 2025-02-15 15:21:44,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,623 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[27]: [(8192,), int64, CPU] 2025-02-15 15:21:44,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,623 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[28]: [(8192,), int64, CPU] 2025-02-15 15:21:44,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,624 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[29]: [(8192,), int64, CPU] 2025-02-15 15:21:44,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,624 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[30]: [(8192,), int64, CPU] 2025-02-15 15:21:44,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,625 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[31]: [(8192,), int64, CPU] 2025-02-15 15:21:44,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,625 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[32]: [(8192,), int64, CPU] 2025-02-15 15:21:44,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,626 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[33]: [(8192,), int64, CPU] 2025-02-15 15:21:44,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,626 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[34]: [(8192,), int64, CPU] 2025-02-15 15:21:44,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,627 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[35]: [(8192,), int64, CPU] 2025-02-15 15:21:44,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,627 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[36]: [(8192,), int64, CPU] 2025-02-15 15:21:44,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,628 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[37]: [(8192,), int64, CPU] 2025-02-15 15:21:44,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,628 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[38]: [(8192,), int64, CPU] 2025-02-15 15:21:44,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,629 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[39]: [(8192,), int64, CPU] 2025-02-15 15:21:44,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,629 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[40]: [(8192,), int64, CPU] 2025-02-15 15:21:44,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,630 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[41]: [(8192,), int64, CPU] 2025-02-15 15:21:44,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,630 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[42]: [(8192,), int64, CPU] 2025-02-15 15:21:44,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,631 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[43]: [(8192,), int64, CPU] 2025-02-15 15:21:44,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,632 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[44]: [(8192,), int64, CPU] 2025-02-15 15:21:44,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,632 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[45]: [(8192,), int64, CPU] 2025-02-15 15:21:44,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,633 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[46]: [(8192,), int64, CPU] 2025-02-15 15:21:44,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,633 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[47]: [(8192,), int64, CPU] 2025-02-15 15:21:44,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,634 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[48]: [(8192,), int64, CPU] 2025-02-15 15:21:44,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,634 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[49]: [(8192,), int64, CPU] 2025-02-15 15:21:44,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,635 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[50]: [(8192,), int64, CPU] 2025-02-15 15:21:44,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,635 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[51]: [(8192,), int64, CPU] 2025-02-15 15:21:44,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,636 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[52]: [(8192,), int64, CPU] 2025-02-15 15:21:44,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,636 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[53]: [(8192,), int64, CPU] 2025-02-15 15:21:44,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,637 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[54]: [(8192,), int64, CPU] 2025-02-15 15:21:44,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,638 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[55]: [(8192,), int64, CPU] 2025-02-15 15:21:44,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,638 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[56]: [(8192,), int64, CPU] 2025-02-15 15:21:44,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,639 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[57]: [(8192,), int64, CPU] 2025-02-15 15:21:44,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,639 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[58]: [(8192,), int64, CPU] 2025-02-15 15:21:44,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,640 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[59]: [(8192,), int64, CPU] 2025-02-15 15:21:44,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,640 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[60]: [(8192,), int64, CPU] 2025-02-15 15:21:44,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,641 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[61]: [(8192,), int64, CPU] 2025-02-15 15:21:44,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,641 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[62]: [(8192,), int64, CPU] 2025-02-15 15:21:44,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,642 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[63]: [(8192,), int64, CPU] 2025-02-15 15:21:44,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,642 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[64]: [(8192,), int64, CPU] 2025-02-15 15:21:44,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,643 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[65]: [(8192,), int64, CPU] 2025-02-15 15:21:44,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,644 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[66]: [(8192,), int64, CPU] 2025-02-15 15:21:44,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,644 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[67]: [(8192,), int64, CPU] 2025-02-15 15:21:44,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,645 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[68]: [(8192,), int64, CPU] 2025-02-15 15:21:44,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,646 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[69]: [(8192,), int64, CPU] 2025-02-15 15:21:44,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,646 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[70]: [(8192,), int64, CPU] 2025-02-15 15:21:44,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,647 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[71]: [(8192,), int64, CPU] 2025-02-15 15:21:44,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,648 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[72]: [(8192,), int64, CPU] 2025-02-15 15:21:44,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,649 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[73]: [(8192,), int64, CPU] 2025-02-15 15:21:44,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,649 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[74]: [(8192,), int64, CPU] 2025-02-15 15:21:44,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,650 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[75]: [(8192,), int64, CPU] 2025-02-15 15:21:44,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,651 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[76]: [(8192,), int64, CPU] 2025-02-15 15:21:44,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,652 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[77]: [(8192,), int64, CPU] 2025-02-15 15:21:44,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,652 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[78]: [(8192,), int64, CPU] 2025-02-15 15:21:44,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,653 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[79]: [(8192,), int64, CPU] 2025-02-15 15:21:44,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,654 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[80]: [(8192,), int64, CPU] 2025-02-15 15:21:44,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,654 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[81]: [(8192,), int64, CPU] 2025-02-15 15:21:44,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,655 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[82]: [(8192,), int64, CPU] 2025-02-15 15:21:44,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,656 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[83]: [(8192,), int64, CPU] 2025-02-15 15:21:44,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,657 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[84]: [(8192,), int64, CPU] 2025-02-15 15:21:44,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,657 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[85]: [(8192,), int64, CPU] 2025-02-15 15:21:44,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,658 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[86]: [(8192,), int64, CPU] 2025-02-15 15:21:44,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,658 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[87]: [(8192,), int64, CPU] 2025-02-15 15:21:44,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,659 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[88]: [(8192,), int64, CPU] 2025-02-15 15:21:44,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,659 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[89]: [(8192,), int64, CPU] 2025-02-15 15:21:44,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,660 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[90]: [(8192,), int64, CPU] 2025-02-15 15:21:44,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,660 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[91]: [(8192,), int64, CPU] 2025-02-15 15:21:44,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,661 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[92]: [(8192,), int64, CPU] 2025-02-15 15:21:44,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,661 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[93]: [(8192,), int64, CPU] 2025-02-15 15:21:44,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,662 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[94]: [(8192,), int64, CPU] 2025-02-15 15:21:44,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,662 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[95]: [(8192,), int64, CPU] 2025-02-15 15:21:44,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,663 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[96]: [(8192,), int64, CPU] 2025-02-15 15:21:44,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,664 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[97]: [(8192,), int64, CPU] 2025-02-15 15:21:44,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,664 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[98]: [(8192,), int64, CPU] 2025-02-15 15:21:44,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,665 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[99]: [(8192,), int64, CPU] 2025-02-15 15:21:44,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,665 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[100]: [(8192,), int64, CPU] 2025-02-15 15:21:44,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,666 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[101]: [(8192,), int64, CPU] 2025-02-15 15:21:44,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,666 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[102]: [(8192,), int64, CPU] 2025-02-15 15:21:44,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,667 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[103]: [(8192,), int64, CPU] 2025-02-15 15:21:44,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,667 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[104]: [(8192,), int64, CPU] 2025-02-15 15:21:44,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,668 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[105]: [(8192,), int64, CPU] 2025-02-15 15:21:44,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,668 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[106]: [(8192,), int64, CPU] 2025-02-15 15:21:44,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,669 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[107]: [(8192,), int64, CPU] 2025-02-15 15:21:44,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,669 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[108]: [(8192,), int64, CPU] 2025-02-15 15:21:44,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,670 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[109]: [(8192,), int64, CPU] 2025-02-15 15:21:44,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,671 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[110]: [(8192,), int64, CPU] 2025-02-15 15:21:44,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,672 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[111]: [(8192,), int64, CPU] 2025-02-15 15:21:44,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,673 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[112]: [(8192,), int64, CPU] 2025-02-15 15:21:44,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,673 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[113]: [(8192,), int64, CPU] 2025-02-15 15:21:44,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,674 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[114]: [(8192,), int64, CPU] 2025-02-15 15:21:44,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,675 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[115]: [(8192,), int64, CPU] 2025-02-15 15:21:44,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,675 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[116]: [(8192,), int64, CPU] 2025-02-15 15:21:44,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,676 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[117]: [(8192,), int64, CPU] 2025-02-15 15:21:44,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,677 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[118]: [(8192,), int64, CPU] 2025-02-15 15:21:44,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,678 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[119]: [(8192,), int64, CPU] 2025-02-15 15:21:44,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,678 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[120]: [(8192,), int64, CPU] 2025-02-15 15:21:44,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,679 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[121]: [(8192,), int64, CPU] 2025-02-15 15:21:44,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,680 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[122]: [(8192,), int64, CPU] 2025-02-15 15:21:44,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,681 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[123]: [(8192,), int64, CPU] 2025-02-15 15:21:44,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,681 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[124]: [(8192,), int64, CPU] 2025-02-15 15:21:44,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,682 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[125]: [(8192,), int64, CPU] 2025-02-15 15:21:44,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,683 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[126]: [(8192,), int64, CPU] 2025-02-15 15:21:44,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,683 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[127]: [(8192,), int64, CPU] 2025-02-15 15:21:44,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,684 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[128]: [(8192,), int64, CPU] 2025-02-15 15:21:44,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,684 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[129]: [(8192,), int64, CPU] 2025-02-15 15:21:44,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,685 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[130]: [(8192,), int64, CPU] 2025-02-15 15:21:44,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,685 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[131]: [(8192,), int64, CPU] 2025-02-15 15:21:44,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,686 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[132]: [(8192,), int64, CPU] 2025-02-15 15:21:44,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,686 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[133]: [(8192,), int64, CPU] 2025-02-15 15:21:44,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,687 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[134]: [(8192,), int64, CPU] 2025-02-15 15:21:44,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,688 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[135]: [(8192,), int64, CPU] 2025-02-15 15:21:44,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,688 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[136]: [(8192,), int64, CPU] 2025-02-15 15:21:44,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,689 - resource_logging.py:44 - debug_tensor - DEBUG - inputs[137]: [(8192,), int64, CPU] 2025-02-15 15:21:44,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,689 - resource_logging.py:44 - debug_tensor - DEBUG - masks[0]: [(8192,), bool, CPU] 2025-02-15 15:21:44,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,690 - resource_logging.py:44 - debug_tensor - DEBUG - masks[1]: [(8192,), bool, CPU] 2025-02-15 15:21:44,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,690 - resource_logging.py:44 - debug_tensor - DEBUG - masks[2]: [(8192,), bool, CPU] 2025-02-15 15:21:44,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,691 - resource_logging.py:44 - debug_tensor - DEBUG - masks[3]: [(8192,), bool, CPU] 2025-02-15 15:21:44,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,691 - resource_logging.py:44 - debug_tensor - DEBUG - masks[4]: [(8192,), bool, CPU] 2025-02-15 15:21:44,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,692 - resource_logging.py:44 - debug_tensor - DEBUG - masks[5]: [(8192,), bool, CPU] 2025-02-15 15:21:44,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,692 - resource_logging.py:44 - debug_tensor - DEBUG - masks[6]: [(8192,), bool, CPU] 2025-02-15 15:21:44,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,693 - resource_logging.py:44 - debug_tensor - DEBUG - masks[7]: [(8192,), bool, CPU] 2025-02-15 15:21:44,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,693 - resource_logging.py:44 - debug_tensor - DEBUG - masks[8]: [(8192,), bool, CPU] 2025-02-15 15:21:44,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,694 - resource_logging.py:44 - debug_tensor - DEBUG - masks[9]: [(8192,), bool, CPU] 2025-02-15 15:21:44,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,694 - resource_logging.py:44 - debug_tensor - DEBUG - masks[10]: [(8192,), bool, CPU] 2025-02-15 15:21:44,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,695 - resource_logging.py:44 - debug_tensor - DEBUG - masks[11]: [(8192,), bool, CPU] 2025-02-15 15:21:44,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,695 - resource_logging.py:44 - debug_tensor - DEBUG - masks[12]: [(8192,), bool, CPU] 2025-02-15 15:21:44,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,696 - resource_logging.py:44 - debug_tensor - DEBUG - masks[13]: [(8192,), bool, CPU] 2025-02-15 15:21:44,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,697 - resource_logging.py:44 - debug_tensor - DEBUG - masks[14]: [(8192,), bool, CPU] 2025-02-15 15:21:44,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,697 - resource_logging.py:44 - debug_tensor - DEBUG - masks[15]: [(8192,), bool, CPU] 2025-02-15 15:21:44,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,698 - resource_logging.py:44 - debug_tensor - DEBUG - masks[16]: [(8192,), bool, CPU] 2025-02-15 15:21:44,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,698 - resource_logging.py:44 - debug_tensor - DEBUG - masks[17]: [(8192,), bool, CPU] 2025-02-15 15:21:44,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,699 - resource_logging.py:44 - debug_tensor - DEBUG - masks[18]: [(8192,), bool, CPU] 2025-02-15 15:21:44,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,699 - resource_logging.py:44 - debug_tensor - DEBUG - masks[19]: [(8192,), bool, CPU] 2025-02-15 15:21:44,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,700 - resource_logging.py:44 - debug_tensor - DEBUG - masks[20]: [(8192,), bool, CPU] 2025-02-15 15:21:44,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,700 - resource_logging.py:44 - debug_tensor - DEBUG - masks[21]: [(8192,), bool, CPU] 2025-02-15 15:21:44,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,701 - resource_logging.py:44 - debug_tensor - DEBUG - masks[22]: [(8192,), bool, CPU] 2025-02-15 15:21:44,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,701 - resource_logging.py:44 - debug_tensor - DEBUG - masks[23]: [(8192,), bool, CPU] 2025-02-15 15:21:44,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,702 - resource_logging.py:44 - debug_tensor - DEBUG - masks[24]: [(8192,), bool, CPU] 2025-02-15 15:21:44,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,702 - resource_logging.py:44 - debug_tensor - DEBUG - masks[25]: [(8192,), bool, CPU] 2025-02-15 15:21:44,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,703 - resource_logging.py:44 - debug_tensor - DEBUG - masks[26]: [(8192,), bool, CPU] 2025-02-15 15:21:44,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,703 - resource_logging.py:44 - debug_tensor - DEBUG - masks[27]: [(8192,), bool, CPU] 2025-02-15 15:21:44,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,704 - resource_logging.py:44 - debug_tensor - DEBUG - masks[28]: [(8192,), bool, CPU] 2025-02-15 15:21:44,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,704 - resource_logging.py:44 - debug_tensor - DEBUG - masks[29]: [(8192,), bool, CPU] 2025-02-15 15:21:44,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,705 - resource_logging.py:44 - debug_tensor - DEBUG - masks[30]: [(8192,), bool, CPU] 2025-02-15 15:21:44,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,705 - resource_logging.py:44 - debug_tensor - DEBUG - masks[31]: [(8192,), bool, CPU] 2025-02-15 15:21:44,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,706 - resource_logging.py:44 - debug_tensor - DEBUG - masks[32]: [(8192,), bool, CPU] 2025-02-15 15:21:44,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,706 - resource_logging.py:44 - debug_tensor - DEBUG - masks[33]: [(8192,), bool, CPU] 2025-02-15 15:21:44,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,707 - resource_logging.py:44 - debug_tensor - DEBUG - masks[34]: [(8192,), bool, CPU] 2025-02-15 15:21:44,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,708 - resource_logging.py:44 - debug_tensor - DEBUG - masks[35]: [(8192,), bool, CPU] 2025-02-15 15:21:44,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,708 - resource_logging.py:44 - debug_tensor - DEBUG - masks[36]: [(8192,), bool, CPU] 2025-02-15 15:21:44,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,709 - resource_logging.py:44 - debug_tensor - DEBUG - masks[37]: [(8192,), bool, CPU] 2025-02-15 15:21:44,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,709 - resource_logging.py:44 - debug_tensor - DEBUG - masks[38]: [(8192,), bool, CPU] 2025-02-15 15:21:44,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,710 - resource_logging.py:44 - debug_tensor - DEBUG - masks[39]: [(8192,), bool, CPU] 2025-02-15 15:21:44,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,710 - resource_logging.py:44 - debug_tensor - DEBUG - masks[40]: [(8192,), bool, CPU] 2025-02-15 15:21:44,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,711 - resource_logging.py:44 - debug_tensor - DEBUG - masks[41]: [(8192,), bool, CPU] 2025-02-15 15:21:44,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,711 - resource_logging.py:44 - debug_tensor - DEBUG - masks[42]: [(8192,), bool, CPU] 2025-02-15 15:21:44,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,712 - resource_logging.py:44 - debug_tensor - DEBUG - masks[43]: [(8192,), bool, CPU] 2025-02-15 15:21:44,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,712 - resource_logging.py:44 - debug_tensor - DEBUG - masks[44]: [(8192,), bool, CPU] 2025-02-15 15:21:44,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,713 - resource_logging.py:44 - debug_tensor - DEBUG - masks[45]: [(8192,), bool, CPU] 2025-02-15 15:21:44,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,713 - resource_logging.py:44 - debug_tensor - DEBUG - masks[46]: [(8192,), bool, CPU] 2025-02-15 15:21:44,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,714 - resource_logging.py:44 - debug_tensor - DEBUG - masks[47]: [(8192,), bool, CPU] 2025-02-15 15:21:44,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,714 - resource_logging.py:44 - debug_tensor - DEBUG - masks[48]: [(8192,), bool, CPU] 2025-02-15 15:21:44,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,715 - resource_logging.py:44 - debug_tensor - DEBUG - masks[49]: [(8192,), bool, CPU] 2025-02-15 15:21:44,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,715 - resource_logging.py:44 - debug_tensor - DEBUG - masks[50]: [(8192,), bool, CPU] 2025-02-15 15:21:44,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,716 - resource_logging.py:44 - debug_tensor - DEBUG - masks[51]: [(8192,), bool, CPU] 2025-02-15 15:21:44,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,717 - resource_logging.py:44 - debug_tensor - DEBUG - masks[52]: [(8192,), bool, CPU] 2025-02-15 15:21:44,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,717 - resource_logging.py:44 - debug_tensor - DEBUG - masks[53]: [(8192,), bool, CPU] 2025-02-15 15:21:44,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,718 - resource_logging.py:44 - debug_tensor - DEBUG - masks[54]: [(8192,), bool, CPU] 2025-02-15 15:21:44,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,718 - resource_logging.py:44 - debug_tensor - DEBUG - masks[55]: [(8192,), bool, CPU] 2025-02-15 15:21:44,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,719 - resource_logging.py:44 - debug_tensor - DEBUG - masks[56]: [(8192,), bool, CPU] 2025-02-15 15:21:44,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,719 - resource_logging.py:44 - debug_tensor - DEBUG - masks[57]: [(8192,), bool, CPU] 2025-02-15 15:21:44,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,720 - resource_logging.py:44 - debug_tensor - DEBUG - masks[58]: [(8192,), bool, CPU] 2025-02-15 15:21:44,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,720 - resource_logging.py:44 - debug_tensor - DEBUG - masks[59]: [(8192,), bool, CPU] 2025-02-15 15:21:44,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,721 - resource_logging.py:44 - debug_tensor - DEBUG - masks[60]: [(8192,), bool, CPU] 2025-02-15 15:21:44,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,721 - resource_logging.py:44 - debug_tensor - DEBUG - masks[61]: [(8192,), bool, CPU] 2025-02-15 15:21:44,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,722 - resource_logging.py:44 - debug_tensor - DEBUG - masks[62]: [(8192,), bool, CPU] 2025-02-15 15:21:44,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,722 - resource_logging.py:44 - debug_tensor - DEBUG - masks[63]: [(8192,), bool, CPU] 2025-02-15 15:21:44,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,723 - resource_logging.py:44 - debug_tensor - DEBUG - masks[64]: [(8192,), bool, CPU] 2025-02-15 15:21:44,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,723 - resource_logging.py:44 - debug_tensor - DEBUG - masks[65]: [(8192,), bool, CPU] 2025-02-15 15:21:44,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,724 - resource_logging.py:44 - debug_tensor - DEBUG - masks[66]: [(8192,), bool, CPU] 2025-02-15 15:21:44,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,724 - resource_logging.py:44 - debug_tensor - DEBUG - masks[67]: [(8192,), bool, CPU] 2025-02-15 15:21:44,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,725 - resource_logging.py:44 - debug_tensor - DEBUG - masks[68]: [(8192,), bool, CPU] 2025-02-15 15:21:44,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,725 - resource_logging.py:44 - debug_tensor - DEBUG - masks[69]: [(8192,), bool, CPU] 2025-02-15 15:21:44,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,726 - resource_logging.py:44 - debug_tensor - DEBUG - masks[70]: [(8192,), bool, CPU] 2025-02-15 15:21:44,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,726 - resource_logging.py:44 - debug_tensor - DEBUG - masks[71]: [(8192,), bool, CPU] 2025-02-15 15:21:44,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,727 - resource_logging.py:44 - debug_tensor - DEBUG - masks[72]: [(8192,), bool, CPU] 2025-02-15 15:21:44,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,728 - resource_logging.py:44 - debug_tensor - DEBUG - masks[73]: [(8192,), bool, CPU] 2025-02-15 15:21:44,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,728 - resource_logging.py:44 - debug_tensor - DEBUG - masks[74]: [(8192,), bool, CPU] 2025-02-15 15:21:44,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,729 - resource_logging.py:44 - debug_tensor - DEBUG - masks[75]: [(8192,), bool, CPU] 2025-02-15 15:21:44,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,729 - resource_logging.py:44 - debug_tensor - DEBUG - masks[76]: [(8192,), bool, CPU] 2025-02-15 15:21:44,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,730 - resource_logging.py:44 - debug_tensor - DEBUG - masks[77]: [(8192,), bool, CPU] 2025-02-15 15:21:44,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,730 - resource_logging.py:44 - debug_tensor - DEBUG - masks[78]: [(8192,), bool, CPU] 2025-02-15 15:21:44,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,731 - resource_logging.py:44 - debug_tensor - DEBUG - masks[79]: [(8192,), bool, CPU] 2025-02-15 15:21:44,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,731 - resource_logging.py:44 - debug_tensor - DEBUG - masks[80]: [(8192,), bool, CPU] 2025-02-15 15:21:44,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,732 - resource_logging.py:44 - debug_tensor - DEBUG - masks[81]: [(8192,), bool, CPU] 2025-02-15 15:21:44,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,732 - resource_logging.py:44 - debug_tensor - DEBUG - masks[82]: [(8192,), bool, CPU] 2025-02-15 15:21:44,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,733 - resource_logging.py:44 - debug_tensor - DEBUG - masks[83]: [(8192,), bool, CPU] 2025-02-15 15:21:44,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,733 - resource_logging.py:44 - debug_tensor - DEBUG - masks[84]: [(8192,), bool, CPU] 2025-02-15 15:21:44,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,734 - resource_logging.py:44 - debug_tensor - DEBUG - masks[85]: [(8192,), bool, CPU] 2025-02-15 15:21:44,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,734 - resource_logging.py:44 - debug_tensor - DEBUG - masks[86]: [(8192,), bool, CPU] 2025-02-15 15:21:44,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,735 - resource_logging.py:44 - debug_tensor - DEBUG - masks[87]: [(8192,), bool, CPU] 2025-02-15 15:21:44,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,735 - resource_logging.py:44 - debug_tensor - DEBUG - masks[88]: [(8192,), bool, CPU] 2025-02-15 15:21:44,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,736 - resource_logging.py:44 - debug_tensor - DEBUG - masks[89]: [(8192,), bool, CPU] 2025-02-15 15:21:44,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,736 - resource_logging.py:44 - debug_tensor - DEBUG - masks[90]: [(8192,), bool, CPU] 2025-02-15 15:21:44,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,737 - resource_logging.py:44 - debug_tensor - DEBUG - masks[91]: [(8192,), bool, CPU] 2025-02-15 15:21:44,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,738 - resource_logging.py:44 - debug_tensor - DEBUG - masks[92]: [(8192,), bool, CPU] 2025-02-15 15:21:44,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,738 - resource_logging.py:44 - debug_tensor - DEBUG - masks[93]: [(8192,), bool, CPU] 2025-02-15 15:21:44,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,739 - resource_logging.py:44 - debug_tensor - DEBUG - masks[94]: [(8192,), bool, CPU] 2025-02-15 15:21:44,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,739 - resource_logging.py:44 - debug_tensor - DEBUG - masks[95]: [(8192,), bool, CPU] 2025-02-15 15:21:44,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,740 - resource_logging.py:44 - debug_tensor - DEBUG - masks[96]: [(8192,), bool, CPU] 2025-02-15 15:21:44,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,740 - resource_logging.py:44 - debug_tensor - DEBUG - masks[97]: [(8192,), bool, CPU] 2025-02-15 15:21:44,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,741 - resource_logging.py:44 - debug_tensor - DEBUG - masks[98]: [(8192,), bool, CPU] 2025-02-15 15:21:44,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,741 - resource_logging.py:44 - debug_tensor - DEBUG - masks[99]: [(8192,), bool, CPU] 2025-02-15 15:21:44,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,742 - resource_logging.py:44 - debug_tensor - DEBUG - masks[100]: [(8192,), bool, CPU] 2025-02-15 15:21:44,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,742 - resource_logging.py:44 - debug_tensor - DEBUG - masks[101]: [(8192,), bool, CPU] 2025-02-15 15:21:44,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,743 - resource_logging.py:44 - debug_tensor - DEBUG - masks[102]: [(8192,), bool, CPU] 2025-02-15 15:21:44,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,743 - resource_logging.py:44 - debug_tensor - DEBUG - masks[103]: [(8192,), bool, CPU] 2025-02-15 15:21:44,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,744 - resource_logging.py:44 - debug_tensor - DEBUG - masks[104]: [(8192,), bool, CPU] 2025-02-15 15:21:44,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,744 - resource_logging.py:44 - debug_tensor - DEBUG - masks[105]: [(8192,), bool, CPU] 2025-02-15 15:21:44,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,745 - resource_logging.py:44 - debug_tensor - DEBUG - masks[106]: [(8192,), bool, CPU] 2025-02-15 15:21:44,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,745 - resource_logging.py:44 - debug_tensor - DEBUG - masks[107]: [(8192,), bool, CPU] 2025-02-15 15:21:44,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,746 - resource_logging.py:44 - debug_tensor - DEBUG - masks[108]: [(8192,), bool, CPU] 2025-02-15 15:21:44,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,747 - resource_logging.py:44 - debug_tensor - DEBUG - masks[109]: [(8192,), bool, CPU] 2025-02-15 15:21:44,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,747 - resource_logging.py:44 - debug_tensor - DEBUG - masks[110]: [(8192,), bool, CPU] 2025-02-15 15:21:44,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,748 - resource_logging.py:44 - debug_tensor - DEBUG - masks[111]: [(8192,), bool, CPU] 2025-02-15 15:21:44,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,748 - resource_logging.py:44 - debug_tensor - DEBUG - masks[112]: [(8192,), bool, CPU] 2025-02-15 15:21:44,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,749 - resource_logging.py:44 - debug_tensor - DEBUG - masks[113]: [(8192,), bool, CPU] 2025-02-15 15:21:44,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,749 - resource_logging.py:44 - debug_tensor - DEBUG - masks[114]: [(8192,), bool, CPU] 2025-02-15 15:21:44,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,750 - resource_logging.py:44 - debug_tensor - DEBUG - masks[115]: [(8192,), bool, CPU] 2025-02-15 15:21:44,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,750 - resource_logging.py:44 - debug_tensor - DEBUG - masks[116]: [(8192,), bool, CPU] 2025-02-15 15:21:44,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,751 - resource_logging.py:44 - debug_tensor - DEBUG - masks[117]: [(8192,), bool, CPU] 2025-02-15 15:21:44,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,751 - resource_logging.py:44 - debug_tensor - DEBUG - masks[118]: [(8192,), bool, CPU] 2025-02-15 15:21:44,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,752 - resource_logging.py:44 - debug_tensor - DEBUG - masks[119]: [(8192,), bool, CPU] 2025-02-15 15:21:44,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,752 - resource_logging.py:44 - debug_tensor - DEBUG - masks[120]: [(8192,), bool, CPU] 2025-02-15 15:21:44,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,753 - resource_logging.py:44 - debug_tensor - DEBUG - masks[121]: [(8192,), bool, CPU] 2025-02-15 15:21:44,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,753 - resource_logging.py:44 - debug_tensor - DEBUG - masks[122]: [(8192,), bool, CPU] 2025-02-15 15:21:44,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,754 - resource_logging.py:44 - debug_tensor - DEBUG - masks[123]: [(8192,), bool, CPU] 2025-02-15 15:21:44,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,754 - resource_logging.py:44 - debug_tensor - DEBUG - masks[124]: [(8192,), bool, CPU] 2025-02-15 15:21:44,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,755 - resource_logging.py:44 - debug_tensor - DEBUG - masks[125]: [(8192,), bool, CPU] 2025-02-15 15:21:44,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,755 - resource_logging.py:44 - debug_tensor - DEBUG - masks[126]: [(8192,), bool, CPU] 2025-02-15 15:21:44,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,756 - resource_logging.py:44 - debug_tensor - DEBUG - masks[127]: [(8192,), bool, CPU] 2025-02-15 15:21:44,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,756 - resource_logging.py:44 - debug_tensor - DEBUG - masks[128]: [(8192,), bool, CPU] 2025-02-15 15:21:44,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,757 - resource_logging.py:44 - debug_tensor - DEBUG - masks[129]: [(8192,), bool, CPU] 2025-02-15 15:21:44,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,758 - resource_logging.py:44 - debug_tensor - DEBUG - masks[130]: [(8192,), bool, CPU] 2025-02-15 15:21:44,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,758 - resource_logging.py:44 - debug_tensor - DEBUG - masks[131]: [(8192,), bool, CPU] 2025-02-15 15:21:44,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,759 - resource_logging.py:44 - debug_tensor - DEBUG - masks[132]: [(8192,), bool, CPU] 2025-02-15 15:21:44,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,759 - resource_logging.py:44 - debug_tensor - DEBUG - masks[133]: [(8192,), bool, CPU] 2025-02-15 15:21:44,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,760 - resource_logging.py:44 - debug_tensor - DEBUG - masks[134]: [(8192,), bool, CPU] 2025-02-15 15:21:44,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,760 - resource_logging.py:44 - debug_tensor - DEBUG - masks[135]: [(8192,), bool, CPU] 2025-02-15 15:21:44,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,761 - resource_logging.py:44 - debug_tensor - DEBUG - masks[136]: [(8192,), bool, CPU] 2025-02-15 15:21:44,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,761 - resource_logging.py:44 - debug_tensor - DEBUG - masks[137]: [(8192,), bool, CPU] 2025-02-15 15:21:44,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,769 - resource_logging.py:45 - debug_tensor - DEBUG - preds: [torch.Size([138, 237, 128256]), torch.float32, cpu] 2025-02-15 15:21:44,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,770 - resource_logging.py:45 - debug_tensor - DEBUG - labels: [torch.Size([138, 8192]), torch.int64, cpu] 2025-02-15 15:21:44,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,770 - resource_logging.py:45 - debug_tensor - DEBUG - attention_mask: [torch.Size([138, 8192]), torch.bool, cpu] 2025-02-15 15:21:44,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:21:44,771 - resource_logging.py:45 - debug_tensor - DEBUG - input_ids: [torch.Size([138, 8192]), torch.int64, cpu] 2025-02-15 15:21:44,780 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 0: output_range=[225, 237] 2025-02-15 15:21:44,782 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 0: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,782 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 0: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,782 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 0: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,784 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 1: output_range=[225, 237] 2025-02-15 15:21:44,785 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 1: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,785 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 1: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,785 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 1: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,787 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 2: output_range=[225, 237] 2025-02-15 15:21:44,787 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 2: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,787 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 2: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,787 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 2: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,789 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 3: output_range=[225, 237] 2025-02-15 15:21:44,790 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 3: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,790 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 3: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,790 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 3: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,792 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 4: output_range=[225, 237] 2025-02-15 15:21:44,792 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 4: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,792 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 4: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,792 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 4: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,794 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 5: output_range=[225, 237] 2025-02-15 15:21:44,795 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 5: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,795 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 5: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,795 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 5: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,797 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 6: output_range=[225, 237] 2025-02-15 15:21:44,797 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 6: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,797 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 6: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,797 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 6: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,799 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 7: output_range=[225, 237] 2025-02-15 15:21:44,800 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 7: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,800 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 7: decoded_outputs=['The video rate for this video is 2.'] 2025-02-15 15:21:44,800 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 7: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,801 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 8: output_range=[225, 237] 2025-02-15 15:21:44,802 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 8: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,802 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 8: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,802 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 8: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,804 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 9: output_range=[225, 237] 2025-02-15 15:21:44,804 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 9: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,805 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 9: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,805 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 9: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,806 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 10: output_range=[225, 237] 2025-02-15 15:21:44,807 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 10: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,807 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 10: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,807 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 10: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,809 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 11: output_range=[225, 237] 2025-02-15 15:21:44,809 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 11: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,809 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 11: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,810 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 11: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,811 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 12: output_range=[225, 237] 2025-02-15 15:21:44,812 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 12: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,812 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 12: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,812 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 12: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,814 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 13: output_range=[225, 237] 2025-02-15 15:21:44,814 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 13: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,814 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 13: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,814 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 13: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,816 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 14: output_range=[225, 237] 2025-02-15 15:21:44,817 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 14: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,817 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 14: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,817 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 14: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,819 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 15: output_range=[225, 237] 2025-02-15 15:21:44,819 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 15: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,819 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 15: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,819 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 15: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,821 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 16: output_range=[225, 237] 2025-02-15 15:21:44,822 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 16: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,822 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 16: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,822 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 16: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,824 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 17: output_range=[225, 237] 2025-02-15 15:21:44,824 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 17: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,824 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 17: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,824 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 17: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,826 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 18: output_range=[225, 237] 2025-02-15 15:21:44,827 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 18: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,827 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 18: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,827 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 18: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,829 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 19: output_range=[225, 237] 2025-02-15 15:21:44,829 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 19: cur_outputs=tensor([[ 791, 20392, 4478, 369, 279, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,829 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 19: decoded_outputs=['The engagement rate for the video is 2.'] 2025-02-15 15:21:44,829 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 19: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,831 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 20: output_range=[225, 237] 2025-02-15 15:21:44,832 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 20: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,832 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 20: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,832 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 20: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,833 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 21: output_range=[225, 237] 2025-02-15 15:21:44,834 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 21: cur_outputs=tensor([[ 791, 2835, 4478, 369, 279, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,834 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 21: decoded_outputs=['The video rate for the video is 2.'] 2025-02-15 15:21:44,834 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 21: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,836 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 22: output_range=[225, 237] 2025-02-15 15:21:44,836 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 22: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,837 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 22: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,837 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 22: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,838 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 23: output_range=[225, 237] 2025-02-15 15:21:44,839 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 23: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,839 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 23: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,839 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 23: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,841 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 24: output_range=[225, 237] 2025-02-15 15:21:44,841 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 24: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,841 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 24: decoded_outputs=['The video rate for this video is 2.'] 2025-02-15 15:21:44,841 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 24: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,843 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 25: output_range=[225, 237] 2025-02-15 15:21:44,844 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 25: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,844 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 25: decoded_outputs=['The video rate for this video is 2.'] 2025-02-15 15:21:44,844 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 25: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,846 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 26: output_range=[225, 237] 2025-02-15 15:21:44,846 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 26: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,846 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 26: decoded_outputs=['The video rate for this video is 2.'] 2025-02-15 15:21:44,846 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 26: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,848 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 27: output_range=[225, 237] 2025-02-15 15:21:44,849 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 27: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,849 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 27: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,849 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 27: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,851 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 28: output_range=[225, 237] 2025-02-15 15:21:44,851 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 28: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,851 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 28: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,851 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 28: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,853 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 29: output_range=[225, 237] 2025-02-15 15:21:44,854 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 29: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,854 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 29: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,854 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 29: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,855 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 30: output_range=[225, 237] 2025-02-15 15:21:44,856 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 30: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,856 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 30: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,856 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 30: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,858 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 31: output_range=[225, 237] 2025-02-15 15:21:44,858 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 31: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,858 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 31: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,859 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 31: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,860 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 32: output_range=[225, 237] 2025-02-15 15:21:44,861 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 32: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,861 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 32: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,861 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 32: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,863 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 33: output_range=[225, 237] 2025-02-15 15:21:44,863 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 33: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,863 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 33: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,863 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 33: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,865 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 34: output_range=[225, 237] 2025-02-15 15:21:44,866 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 34: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,866 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 34: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,866 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 34: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,868 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 35: output_range=[225, 237] 2025-02-15 15:21:44,868 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 35: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,868 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 35: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,868 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 35: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,870 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 36: output_range=[225, 237] 2025-02-15 15:21:44,871 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 36: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 16, 320, 128009, 128006]]) 2025-02-15 15:21:44,871 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 36: decoded_outputs=['The final rate for this video is 1 ('] 2025-02-15 15:21:44,871 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 36: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,872 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 37: output_range=[225, 237] 2025-02-15 15:21:44,873 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 37: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,873 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 37: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,873 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 37: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,875 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 38: output_range=[225, 237] 2025-02-15 15:21:44,876 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 38: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,876 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 38: decoded_outputs=['The engagement rate for this video is 2 ('] 2025-02-15 15:21:44,876 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 38: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,877 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 39: output_range=[225, 237] 2025-02-15 15:21:44,878 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 39: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,878 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 39: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,878 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 39: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,880 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 40: output_range=[225, 237] 2025-02-15 15:21:44,880 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 40: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,880 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 40: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,881 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 40: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,882 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 41: output_range=[225, 237] 2025-02-15 15:21:44,883 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 41: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,883 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 41: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,883 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 41: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,885 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 42: output_range=[225, 237] 2025-02-15 15:21:44,885 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 42: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,885 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 42: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,885 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 42: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,887 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 43: output_range=[225, 237] 2025-02-15 15:21:44,888 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 43: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,888 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 43: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,888 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 43: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,890 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 44: output_range=[225, 237] 2025-02-15 15:21:44,890 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 44: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,890 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 44: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,890 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 44: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,892 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 45: output_range=[225, 237] 2025-02-15 15:21:44,893 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 45: cur_outputs=tensor([[ 791, 2835, 4478, 369, 279, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,893 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 45: decoded_outputs=['The video rate for the video is 2 ('] 2025-02-15 15:21:44,893 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 45: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,895 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 46: output_range=[225, 237] 2025-02-15 15:21:44,895 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 46: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,895 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 46: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,895 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 46: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,897 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 47: output_range=[225, 237] 2025-02-15 15:21:44,901 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 47: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,901 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 47: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,901 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 47: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,902 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 48: output_range=[225, 237] 2025-02-15 15:21:44,903 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 48: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,903 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 48: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,903 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 48: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,905 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 49: output_range=[225, 237] 2025-02-15 15:21:44,906 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 49: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,906 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 49: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,906 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 49: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,907 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 50: output_range=[225, 237] 2025-02-15 15:21:44,908 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 50: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,908 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 50: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,908 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 50: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,910 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 51: output_range=[225, 237] 2025-02-15 15:21:44,911 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 51: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,911 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 51: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,911 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 51: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,913 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 52: output_range=[225, 237] 2025-02-15 15:21:44,913 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 52: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,913 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 52: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,913 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 52: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,915 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 53: output_range=[225, 237] 2025-02-15 15:21:44,916 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 53: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,916 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 53: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,916 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 53: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,918 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 54: output_range=[225, 237] 2025-02-15 15:21:44,918 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 54: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,918 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 54: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,918 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 54: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,920 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 55: output_range=[225, 237] 2025-02-15 15:21:44,921 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 55: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,921 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 55: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,921 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 55: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,922 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 56: output_range=[225, 237] 2025-02-15 15:21:44,923 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 56: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,923 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 56: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,923 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 56: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,925 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 57: output_range=[225, 237] 2025-02-15 15:21:44,925 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 57: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,925 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 57: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,925 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 57: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,927 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 58: output_range=[225, 237] 2025-02-15 15:21:44,928 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 58: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,928 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 58: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,928 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 58: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,930 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 59: output_range=[225, 237] 2025-02-15 15:21:44,930 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 59: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,930 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 59: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,930 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 59: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,932 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 60: output_range=[225, 237] 2025-02-15 15:21:44,933 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 60: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 16, 320, 128009, 128006]]) 2025-02-15 15:21:44,933 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 60: decoded_outputs=['The final rate for this video is 1 ('] 2025-02-15 15:21:44,933 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 60: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,934 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 61: output_range=[225, 237] 2025-02-15 15:21:44,935 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 61: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,935 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 61: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,935 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 61: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,937 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 62: output_range=[225, 237] 2025-02-15 15:21:44,937 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 62: cur_outputs=tensor([[ 791, 1620, 4478, 369, 279, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,938 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 62: decoded_outputs=['The final rate for the video is 2.'] 2025-02-15 15:21:44,938 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 62: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,939 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 63: output_range=[225, 237] 2025-02-15 15:21:44,940 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 63: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,940 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 63: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:44,940 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 63: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,942 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 64: output_range=[225, 237] 2025-02-15 15:21:44,942 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 64: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,942 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 64: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,942 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 64: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,944 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 65: output_range=[225, 237] 2025-02-15 15:21:44,945 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 65: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,945 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 65: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,945 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 65: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,947 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 66: output_range=[225, 237] 2025-02-15 15:21:44,947 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 66: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,947 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 66: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,947 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 66: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,949 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 67: output_range=[225, 237] 2025-02-15 15:21:44,950 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 67: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,950 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 67: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:44,950 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 67: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,951 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 68: output_range=[225, 237] 2025-02-15 15:21:44,952 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 68: cur_outputs=tensor([[ 791, 1620, 4478, 369, 279, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,952 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 68: decoded_outputs=['The final rate for the video is 2.'] 2025-02-15 15:21:44,952 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 68: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,954 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 69: output_range=[225, 237] 2025-02-15 15:21:44,955 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 69: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,955 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 69: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,955 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 69: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,956 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 70: output_range=[225, 237] 2025-02-15 15:21:44,957 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 70: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,957 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 70: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,957 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 70: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,959 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 71: output_range=[225, 237] 2025-02-15 15:21:44,959 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 71: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,959 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 71: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,960 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 71: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:44,961 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 72: output_range=[225, 237] 2025-02-15 15:21:44,962 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 72: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,962 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 72: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,962 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 72: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,964 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 73: output_range=[225, 237] 2025-02-15 15:21:44,964 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 73: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,964 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 73: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,964 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 73: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,966 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 74: output_range=[225, 237] 2025-02-15 15:21:44,967 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 74: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,967 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 74: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,967 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 74: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,969 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 75: output_range=[225, 237] 2025-02-15 15:21:44,969 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 75: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,969 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 75: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,969 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 75: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,971 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 76: output_range=[225, 237] 2025-02-15 15:21:44,972 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 76: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,972 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 76: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,972 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 76: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,974 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 77: output_range=[225, 237] 2025-02-15 15:21:44,974 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 77: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,974 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 77: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,974 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 77: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,976 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 78: output_range=[225, 237] 2025-02-15 15:21:44,976 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 78: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,977 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 78: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,977 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 78: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,978 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 79: output_range=[225, 237] 2025-02-15 15:21:44,979 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 79: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,979 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 79: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,979 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 79: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,981 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 80: output_range=[225, 237] 2025-02-15 15:21:44,981 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 80: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,981 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 80: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,981 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 80: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,983 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 81: output_range=[225, 237] 2025-02-15 15:21:44,984 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 81: cur_outputs=tensor([[ 791, 1620, 4478, 369, 279, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,984 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 81: decoded_outputs=['The final rate for the video is 2.'] 2025-02-15 15:21:44,984 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 81: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,986 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 82: output_range=[225, 237] 2025-02-15 15:21:44,986 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 82: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,986 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 82: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,986 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 82: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,988 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 83: output_range=[225, 237] 2025-02-15 15:21:44,989 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 83: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:44,989 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 83: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:44,989 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 83: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:44,991 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 84: output_range=[225, 237] 2025-02-15 15:21:44,991 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 84: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:44,991 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 84: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:44,991 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 84: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:44,993 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 85: output_range=[225, 237] 2025-02-15 15:21:45,000 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 85: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,001 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 85: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:45,001 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 85: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,002 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 86: output_range=[225, 237] 2025-02-15 15:21:45,003 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 86: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,003 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 86: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,003 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 86: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,005 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 87: output_range=[225, 237] 2025-02-15 15:21:45,006 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 87: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,006 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 87: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:45,006 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 87: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,007 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 88: output_range=[225, 237] 2025-02-15 15:21:45,008 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 88: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,008 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 88: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,008 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 88: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:45,010 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 89: output_range=[225, 237] 2025-02-15 15:21:45,011 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 89: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,011 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 89: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,011 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 89: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,013 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 90: output_range=[225, 237] 2025-02-15 15:21:45,014 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 90: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,014 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 90: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,014 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 90: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,017 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 91: output_range=[225, 237] 2025-02-15 15:21:45,017 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 91: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,017 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 91: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,018 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 91: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,020 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 92: output_range=[225, 237] 2025-02-15 15:21:45,021 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 92: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,021 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 92: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,021 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 92: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,023 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 93: output_range=[225, 237] 2025-02-15 15:21:45,023 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 93: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,023 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 93: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,023 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 93: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,025 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 94: output_range=[225, 237] 2025-02-15 15:21:45,026 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 94: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,026 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 94: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,026 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 94: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,028 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 95: output_range=[225, 237] 2025-02-15 15:21:45,028 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 95: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,028 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 95: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,029 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 95: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,030 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 96: output_range=[225, 237] 2025-02-15 15:21:45,031 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 96: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,031 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 96: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:45,031 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 96: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,033 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 97: output_range=[225, 237] 2025-02-15 15:21:45,033 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 97: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,033 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 97: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,033 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 97: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,035 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 98: output_range=[225, 237] 2025-02-15 15:21:45,036 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 98: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,036 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 98: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,036 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 98: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,038 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 99: output_range=[225, 237] 2025-02-15 15:21:45,038 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 99: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,038 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 99: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,038 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 99: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,040 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 100: output_range=[225, 237] 2025-02-15 15:21:45,041 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 100: cur_outputs=tensor([[ 791, 2835, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,041 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 100: decoded_outputs=['The video rate for this video is 2 ('] 2025-02-15 15:21:45,041 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 100: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,042 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 101: output_range=[225, 237] 2025-02-15 15:21:45,043 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 101: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,043 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 101: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,043 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 101: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,045 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 102: output_range=[225, 237] 2025-02-15 15:21:45,045 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 102: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,046 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 102: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,046 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 102: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,047 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 103: output_range=[225, 237] 2025-02-15 15:21:45,048 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 103: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,048 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 103: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:45,048 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 103: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,050 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 104: output_range=[225, 237] 2025-02-15 15:21:45,050 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 104: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,050 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 104: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,050 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 104: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,052 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 105: output_range=[225, 237] 2025-02-15 15:21:45,053 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 105: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,053 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 105: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:45,053 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 105: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,055 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 106: output_range=[225, 237] 2025-02-15 15:21:45,055 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 106: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,055 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 106: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,055 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 106: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,057 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 107: output_range=[225, 237] 2025-02-15 15:21:45,058 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 107: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,058 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 107: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,058 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 107: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,059 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 108: output_range=[225, 237] 2025-02-15 15:21:45,060 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 108: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,060 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 108: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,060 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 108: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,062 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 109: output_range=[225, 237] 2025-02-15 15:21:45,062 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 109: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,062 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 109: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,063 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 109: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,064 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 110: output_range=[225, 237] 2025-02-15 15:21:45,065 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 110: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,065 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 110: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,065 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 110: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,067 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 111: output_range=[225, 237] 2025-02-15 15:21:45,067 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 111: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,067 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 111: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,067 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 111: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,069 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 112: output_range=[225, 237] 2025-02-15 15:21:45,070 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 112: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,070 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 112: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,070 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 112: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,072 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 113: output_range=[225, 237] 2025-02-15 15:21:45,072 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 113: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 16, 320, 128009, 128006]]) 2025-02-15 15:21:45,072 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 113: decoded_outputs=['The final rate for this video is 1 ('] 2025-02-15 15:21:45,072 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 113: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,074 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 114: output_range=[225, 237] 2025-02-15 15:21:45,075 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 114: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,075 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 114: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,075 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 114: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,077 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 115: output_range=[225, 237] 2025-02-15 15:21:45,077 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 115: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,077 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 115: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,077 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 115: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,079 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 116: output_range=[225, 237] 2025-02-15 15:21:45,079 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 116: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,080 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 116: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,080 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 116: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,081 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 117: output_range=[225, 237] 2025-02-15 15:21:45,082 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 117: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,082 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 117: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,082 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 117: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,084 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 118: output_range=[225, 237] 2025-02-15 15:21:45,084 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 118: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,084 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 118: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,084 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 118: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,086 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 119: output_range=[225, 237] 2025-02-15 15:21:45,087 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 119: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,087 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 119: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,087 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 119: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,089 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 120: output_range=[225, 237] 2025-02-15 15:21:45,089 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 120: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,089 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 120: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,089 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 120: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,091 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 121: output_range=[225, 237] 2025-02-15 15:21:45,092 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 121: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,092 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 121: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,092 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 121: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,102 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 122: output_range=[225, 237] 2025-02-15 15:21:45,103 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 122: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,103 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 122: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,103 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 122: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,105 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 123: output_range=[225, 237] 2025-02-15 15:21:45,106 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 123: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,106 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 123: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,106 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 123: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,109 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 124: output_range=[225, 237] 2025-02-15 15:21:45,110 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 124: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,110 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 124: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,110 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 124: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,114 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 125: output_range=[225, 237] 2025-02-15 15:21:45,114 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 125: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,115 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 125: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,115 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 125: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,117 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 126: output_range=[225, 237] 2025-02-15 15:21:45,118 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 126: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,118 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 126: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,118 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 126: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,120 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 127: output_range=[225, 237] 2025-02-15 15:21:45,120 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 127: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 16, 320, 128009, 128006]]) 2025-02-15 15:21:45,121 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 127: decoded_outputs=['The final rate for this video is 1 ('] 2025-02-15 15:21:45,121 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 127: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,122 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 128: output_range=[225, 237] 2025-02-15 15:21:45,123 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 128: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,123 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 128: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,123 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 128: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,125 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 129: output_range=[225, 237] 2025-02-15 15:21:45,125 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 129: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,125 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 129: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,125 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 129: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,127 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 130: output_range=[225, 237] 2025-02-15 15:21:45,128 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 130: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,128 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 130: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,128 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 130: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,130 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 131: output_range=[225, 237] 2025-02-15 15:21:45,130 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 131: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,130 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 131: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,130 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 131: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,132 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 132: output_range=[225, 237] 2025-02-15 15:21:45,133 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 132: cur_outputs=tensor([[ 791, 20392, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,133 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 132: decoded_outputs=['The engagement rate for this video is 2.'] 2025-02-15 15:21:45,133 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 132: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,134 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 133: output_range=[225, 237] 2025-02-15 15:21:45,135 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 133: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 13, 128009, 128006]]) 2025-02-15 15:21:45,135 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 133: decoded_outputs=['The final rate for this video is 2.'] 2025-02-15 15:21:45,135 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 133: decoded_labels=['\n\nThe engagement label of the video is 0.'] 2025-02-15 15:21:45,137 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 134: output_range=[225, 237] 2025-02-15 15:21:45,137 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 134: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,138 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 134: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,138 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 134: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,139 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 135: output_range=[225, 237] 2025-02-15 15:21:45,140 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 135: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,140 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 135: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,140 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 135: decoded_labels=['\n\nThe engagement label of the video is 1.'] 2025-02-15 15:21:45,142 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 136: output_range=[225, 237] 2025-02-15 15:21:45,142 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 136: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,142 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 136: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,143 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 136: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,144 - finetune_llama.py:501 - compute_metrics - DEBUG - batch 137: output_range=[225, 237] 2025-02-15 15:21:45,145 - finetune_llama.py:504 - compute_metrics - DEBUG - batch 137: cur_outputs=tensor([[ 791, 1620, 4478, 369, 420, 2835, 374, 220, 17, 320, 128009, 128006]]) 2025-02-15 15:21:45,145 - finetune_llama.py:507 - compute_metrics - DEBUG - batch 137: decoded_outputs=['The final rate for this video is 2 ('] 2025-02-15 15:21:45,145 - finetune_llama.py:509 - compute_metrics - DEBUG - batch 137: decoded_labels=['\n\nThe engagement label of the video is 2.'] 2025-02-15 15:21:45,145 - finetune_llama.py:518 - compute_metrics - DEBUG - pred_labels=[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] 2025-02-15 15:21:45,145 - finetune_llama.py:519 - compute_metrics - DEBUG - gold_labels=[0, 1, 0, 0, 0, 2, 2, 0, 2, 2, 0, 0, 2, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1, 2, 0, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2, 1, 2, 1, 2, 2, 2, 1, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1, 2, 2, 0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 2, 2, 2, 0, 1, 2, 2, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2, 2, 2, 0, 2, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 2, 1, 2, 2] 2025-02-15 15:22:14,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:14,975 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:22:14,980 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:22:14,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:14,984 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 294, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:22:14,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:14,985 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 294, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:22:19,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:22:19,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:22:19,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.57 seconds 2025-02-15 15:22:19,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25121.17 MB 2025-02-15 15:22:19,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26161.62 MB 2025-02-15 15:22:19,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1040.45 MB 2025-02-15 15:22:19,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82862.67 MB 2025-02-15 15:22:19,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -49874.47 MB 2025-02-15 15:22:19,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.52 MB 2025-02-15 15:22:19,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:22:19,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:22:19,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:22:19,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26161.62 MB 2025-02-15 15:22:19,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25064.46 MB 2025-02-15 15:22:19,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1097.15 MB 2025-02-15 15:22:19,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:19,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:19,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27088.72 MB 2025-02-15 15:22:19,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:22:19,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:22:19,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.33 seconds 2025-02-15 15:22:19,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25064.46 MB 2025-02-15 15:22:19,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25152.05 MB 2025-02-15 15:22:19,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 87.59 MB 2025-02-15 15:22:19,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:19,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:19,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29149.18 MB 2025-02-15 15:22:19,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:22:19,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:22:19,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:22:19,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25151.99 MB 2025-02-15 15:22:19,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25463.68 MB 2025-02-15 15:22:19,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.70 MB 2025-02-15 15:22:19,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:19,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:19,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25697.57 MB 2025-02-15 15:22:19,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:22:19,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:22:19,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 15:22:19,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25463.68 MB 2025-02-15 15:22:19,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25842.30 MB 2025-02-15 15:22:19,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.61 MB 2025-02-15 15:22:19,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:19,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:19,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26748.40 MB 2025-02-15 15:22:19,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:22:19,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:22:19,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:22:19,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:19,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25151.99 MB 2025-02-15 15:22:19,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25842.30 MB 2025-02-15 15:22:19,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.31 MB 2025-02-15 15:22:19,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:19,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:22:19,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:19,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26748.40 MB 2025-02-15 15:22:20,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:22:20,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:22:20,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 15:22:20,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:20,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26207.89 MB 2025-02-15 15:22:20,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26366.89 MB 2025-02-15 15:22:20,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.00 MB 2025-02-15 15:22:20,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:22:20,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-15 15:22:20,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 98.57 MB 2025-02-15 15:22:20,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26483.67 MB 2025-02-15 15:22:20,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:22:20,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:22:20,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:22:20,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:20,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26467.47 MB 2025-02-15 15:22:20,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26627.08 MB 2025-02-15 15:22:20,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.62 MB 2025-02-15 15:22:20,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33086.77 MB 2025-02-15 15:22:20,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-15 15:22:20,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:20,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26627.08 MB 2025-02-15 15:22:20,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:22:20,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:22:20,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.04 seconds 2025-02-15 15:22:20,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:20,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24096.85 MB 2025-02-15 15:22:20,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26769.26 MB 2025-02-15 15:22:20,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2672.41 MB 2025-02-15 15:22:20,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82862.67 MB 2025-02-15 15:22:20,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-15 15:22:20,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -49775.90 MB 2025-02-15 15:22:20,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26769.26 MB 2025-02-15 15:22:20,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:22:20,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:22:20,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 15:22:20,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:20,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26769.26 MB 2025-02-15 15:22:20,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26612.61 MB 2025-02-15 15:22:20,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -156.64 MB 2025-02-15 15:22:20,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33086.77 MB 2025-02-15 15:22:20,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-15 15:22:20,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:22:20,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28687.25 MB 2025-02-15 15:22:20,222 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5767, cut from 5769 2025-02-15 15:22:20,223 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:22:20,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:22:20,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:22:20,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:22:20,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:22:20,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26612.61 MB 2025-02-15 15:22:20,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32579.20 MB 2025-02-15 15:22:20,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5966.59 MB 2025-02-15 15:22:20,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33086.77 MB 2025-02-15 15:22:20,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36054.24 MB 2025-02-15 15:22:20,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2967.47 MB 2025-02-15 15:22:20,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32579.20 MB 2025-02-15 15:22:20,340 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5559] 2025-02-15 15:22:20,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:20,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:22:20,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:20,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:22:20,347 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:22:20,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:22:20,348 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:22:20,348 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:23:25,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:25,488 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:23:25,493 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:23:25,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:25,497 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 287, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:23:25,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:25,498 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 287, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:23:29,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:23:29,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:23:29,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.38 seconds 2025-02-15 15:23:29,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:29,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25072.39 MB 2025-02-15 15:23:29,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26088.07 MB 2025-02-15 15:23:29,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1015.68 MB 2025-02-15 15:23:29,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41984.98 MB 2025-02-15 15:23:29,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:23:29,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8996.78 MB 2025-02-15 15:23:29,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34996.75 MB 2025-02-15 15:23:29,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:23:29,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:23:29,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:23:29,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:29,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26088.07 MB 2025-02-15 15:23:29,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26432.61 MB 2025-02-15 15:23:29,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.54 MB 2025-02-15 15:23:29,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:23:29,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:23:29,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:23:29,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29824.32 MB 2025-02-15 15:23:31,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:23:31,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:23:31,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.27 seconds 2025-02-15 15:23:31,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26432.61 MB 2025-02-15 15:23:31,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26785.62 MB 2025-02-15 15:23:31,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.01 MB 2025-02-15 15:23:31,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:23:31,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:23:31,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:23:31,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30772.13 MB 2025-02-15 15:23:31,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:23:31,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:23:31,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:23:31,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26785.62 MB 2025-02-15 15:23:31,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28041.91 MB 2025-02-15 15:23:31,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.28 MB 2025-02-15 15:23:31,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:23:31,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32988.20 MB 2025-02-15 15:23:31,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:23:31,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28984.50 MB 2025-02-15 15:23:31,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:23:31,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:23:31,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 15:23:31,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28041.91 MB 2025-02-15 15:23:31,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29532.76 MB 2025-02-15 15:23:31,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1490.86 MB 2025-02-15 15:23:31,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:23:31,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-15 15:23:31,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:23:31,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33221.26 MB 2025-02-15 15:23:31,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:23:31,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:23:31,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 15:23:31,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26785.62 MB 2025-02-15 15:23:31,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29532.76 MB 2025-02-15 15:23:31,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2747.14 MB 2025-02-15 15:23:31,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32988.20 MB 2025-02-15 15:23:31,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-15 15:23:31,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:23:31,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33221.26 MB 2025-02-15 15:23:31,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:23:31,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:23:31,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 15:23:31,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30552.57 MB 2025-02-15 15:23:31,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31064.20 MB 2025-02-15 15:23:31,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 511.63 MB 2025-02-15 15:23:31,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34875.64 MB 2025-02-15 15:23:31,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35150.36 MB 2025-02-15 15:23:31,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 274.73 MB 2025-02-15 15:23:31,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31534.88 MB 2025-02-15 15:23:31,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:23:31,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:23:31,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:23:31,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31338.77 MB 2025-02-15 15:23:31,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31544.06 MB 2025-02-15 15:23:31,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.28 MB 2025-02-15 15:23:31,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35150.36 MB 2025-02-15 15:23:31,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35154.56 MB 2025-02-15 15:23:31,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 15:23:31,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31605.36 MB 2025-02-15 15:23:31,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:23:31,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:23:31,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.00 seconds 2025-02-15 15:23:31,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24072.46 MB 2025-02-15 15:23:31,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31745.13 MB 2025-02-15 15:23:31,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7672.67 MB 2025-02-15 15:23:31,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41984.98 MB 2025-02-15 15:23:31,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35154.56 MB 2025-02-15 15:23:31,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6830.42 MB 2025-02-15 15:23:31,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31745.13 MB 2025-02-15 15:23:31,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:23:31,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:23:31,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:23:31,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31745.13 MB 2025-02-15 15:23:31,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34759.16 MB 2025-02-15 15:23:31,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 15:23:31,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35154.56 MB 2025-02-15 15:23:31,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36094.08 MB 2025-02-15 15:23:31,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-15 15:23:31,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35060.79 MB 2025-02-15 15:23:31,781 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:23:31,781 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:23:31,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:23:31,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:23:31,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:23:31,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:23:31,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28446.04 MB 2025-02-15 15:23:31,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36885.06 MB 2025-02-15 15:23:31,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:23:31,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36094.08 MB 2025-02-15 15:23:31,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44484.79 MB 2025-02-15 15:23:31,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 15:23:31,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36885.06 MB 2025-02-15 15:23:31,946 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:23:31,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:31,947 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:23:31,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:31,948 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:23:31,953 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:23:31,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:23:31,954 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:23:31,954 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:24:32,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:24:32,797 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:24:32,802 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:24:32,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:24:32,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1538, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:24:32,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:24:32,807 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1538, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:24:56,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:24:56,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:24:56,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.80 seconds 2025-02-15 15:24:56,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:56,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33789.56 MB 2025-02-15 15:24:56,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39232.46 MB 2025-02-15 15:24:56,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5442.90 MB 2025-02-15 15:24:56,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57069.80 MB 2025-02-15 15:24:56,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50071.60 MB 2025-02-15 15:24:56,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6998.20 MB 2025-02-15 15:24:56,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48243.76 MB 2025-02-15 15:24:56,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:24:56,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:24:56,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 15:24:56,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:56,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39232.46 MB 2025-02-15 15:24:56,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33877.26 MB 2025-02-15 15:24:56,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5355.20 MB 2025-02-15 15:24:56,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50071.60 MB 2025-02-15 15:24:56,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55801.02 MB 2025-02-15 15:24:56,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5729.42 MB 2025-02-15 15:24:56,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51112.34 MB 2025-02-15 15:24:58,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:24:58,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:24:58,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:24:58,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:58,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33877.26 MB 2025-02-15 15:24:58,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34408.10 MB 2025-02-15 15:24:58,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:24:58,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55801.02 MB 2025-02-15 15:24:58,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 15:24:58,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11173.63 MB 2025-02-15 15:24:58,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38387.30 MB 2025-02-15 15:24:58,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:24:58,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:24:58,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:24:58,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:58,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34408.10 MB 2025-02-15 15:24:58,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36297.63 MB 2025-02-15 15:24:58,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:24:58,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 15:24:58,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44627.39 MB 2025-02-15 15:24:58,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:24:58,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37715.06 MB 2025-02-15 15:24:58,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:24:58,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:24:58,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:24:58,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:58,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36297.63 MB 2025-02-15 15:24:58,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38539.49 MB 2025-02-15 15:24:58,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:24:58,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 15:24:58,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46514.83 MB 2025-02-15 15:24:58,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:24:58,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44083.77 MB 2025-02-15 15:24:58,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:24:58,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:24:58,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 15:24:58,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:24:58,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34408.10 MB 2025-02-15 15:24:58,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38539.49 MB 2025-02-15 15:24:58,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:24:58,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44627.39 MB 2025-02-15 15:24:58,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46514.83 MB 2025-02-15 15:24:58,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:24:58,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44083.77 MB 2025-02-15 15:25:00,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:25:00,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:25:00,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.33 seconds 2025-02-15 15:25:00,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:25:00,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40073.03 MB 2025-02-15 15:25:00,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30736.21 MB 2025-02-15 15:25:00,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9336.82 MB 2025-02-15 15:25:00,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46514.83 MB 2025-02-15 15:25:00,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46930.07 MB 2025-02-15 15:25:00,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:25:00,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41048.67 MB 2025-02-15 15:25:00,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:25:00,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:25:00,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:25:00,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:25:00,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31149.10 MB 2025-02-15 15:25:00,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31377.12 MB 2025-02-15 15:25:00,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-15 15:25:00,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46930.07 MB 2025-02-15 15:25:00,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46930.07 MB 2025-02-15 15:25:00,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:25:00,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31613.18 MB 2025-02-15 15:25:00,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:25:00,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:25:00,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.48 seconds 2025-02-15 15:25:00,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:25:00,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28431.04 MB 2025-02-15 15:25:00,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31577.07 MB 2025-02-15 15:25:00,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3146.02 MB 2025-02-15 15:25:00,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57069.80 MB 2025-02-15 15:25:00,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46930.07 MB 2025-02-15 15:25:00,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10139.73 MB 2025-02-15 15:25:00,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31613.18 MB 2025-02-15 15:25:00,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:25:00,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:25:00,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:25:00,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:25:00,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31577.07 MB 2025-02-15 15:25:00,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23314.09 MB 2025-02-15 15:25:00,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8262.98 MB 2025-02-15 15:25:00,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46930.07 MB 2025-02-15 15:25:00,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46930.07 MB 2025-02-15 15:25:00,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:25:00,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31577.07 MB 2025-02-15 15:25:00,577 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-15 15:25:00,577 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:25:00,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:25:00,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:25:00,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:25:00,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:25:00,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23314.09 MB 2025-02-15 15:25:00,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31705.30 MB 2025-02-15 15:25:00,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.21 MB 2025-02-15 15:25:00,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46930.07 MB 2025-02-15 15:25:00,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46930.07 MB 2025-02-15 15:25:00,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:25:00,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31705.30 MB 2025-02-15 15:25:00,741 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-15 15:25:00,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:00,743 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:25:00,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:00,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:25:00,749 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:25:00,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:00,750 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:25:00,750 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:25:44,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:44,087 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:25:44,092 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:25:44,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:44,096 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1627, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:25:44,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:25:44,098 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1627, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:26:09,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:26:09,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:26:09,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.35 seconds 2025-02-15 15:26:09,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:09,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24305.90 MB 2025-02-15 15:26:09,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30064.68 MB 2025-02-15 15:26:09,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5758.78 MB 2025-02-15 15:26:09,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51101.30 MB 2025-02-15 15:26:09,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38992.35 MB 2025-02-15 15:26:09,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12108.96 MB 2025-02-15 15:26:09,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38986.60 MB 2025-02-15 15:26:09,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:26:09,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:26:09,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 15:26:09,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:09,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30064.68 MB 2025-02-15 15:26:09,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24236.12 MB 2025-02-15 15:26:09,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5828.57 MB 2025-02-15 15:26:09,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38992.35 MB 2025-02-15 15:26:09,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45359.30 MB 2025-02-15 15:26:09,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6366.95 MB 2025-02-15 15:26:09,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40905.94 MB 2025-02-15 15:26:11,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:26:11,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:26:11,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 15:26:11,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24236.12 MB 2025-02-15 15:26:11,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24766.96 MB 2025-02-15 15:26:11,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:26:11,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45359.30 MB 2025-02-15 15:26:11,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30454.84 MB 2025-02-15 15:26:11,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14904.46 MB 2025-02-15 15:26:11,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28745.50 MB 2025-02-15 15:26:11,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:26:11,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:26:11,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:26:11,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-15 15:26:11,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26656.49 MB 2025-02-15 15:26:11,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:26:11,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30454.84 MB 2025-02-15 15:26:11,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30454.84 MB 2025-02-15 15:26:11,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:26:11,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28073.92 MB 2025-02-15 15:26:11,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:26:11,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:26:11,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:26:11,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26656.49 MB 2025-02-15 15:26:11,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-15 15:26:11,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:26:11,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30454.84 MB 2025-02-15 15:26:11,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36589.01 MB 2025-02-15 15:26:11,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:26:11,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-15 15:26:11,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:26:11,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:26:11,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:26:11,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-15 15:26:11,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-15 15:26:11,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:26:11,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30454.84 MB 2025-02-15 15:26:11,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36589.01 MB 2025-02-15 15:26:11,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:26:11,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-15 15:26:11,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:26:11,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:26:11,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:26:11,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30431.89 MB 2025-02-15 15:26:11,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31198.89 MB 2025-02-15 15:26:11,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:26:11,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36589.01 MB 2025-02-15 15:26:11,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37004.25 MB 2025-02-15 15:26:11,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:26:11,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31906.68 MB 2025-02-15 15:26:11,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:26:11,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:26:11,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:26:11,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31611.78 MB 2025-02-15 15:26:11,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31843.43 MB 2025-02-15 15:26:11,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.65 MB 2025-02-15 15:26:11,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37004.25 MB 2025-02-15 15:26:11,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37004.25 MB 2025-02-15 15:26:11,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:26:11,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32051.50 MB 2025-02-15 15:26:11,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:26:11,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:26:11,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.84 seconds 2025-02-15 15:26:11,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:11,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18637.30 MB 2025-02-15 15:26:11,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32044.51 MB 2025-02-15 15:26:11,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13407.20 MB 2025-02-15 15:26:11,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51101.30 MB 2025-02-15 15:26:11,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37004.25 MB 2025-02-15 15:26:11,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14097.06 MB 2025-02-15 15:26:11,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32051.50 MB 2025-02-15 15:26:12,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:26:12,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:26:12,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:26:12,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:12,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32044.51 MB 2025-02-15 15:26:12,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23641.69 MB 2025-02-15 15:26:12,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8402.81 MB 2025-02-15 15:26:12,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37004.25 MB 2025-02-15 15:26:12,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37004.25 MB 2025-02-15 15:26:12,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:26:12,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.37 MB 2025-02-15 15:26:12,274 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:26:12,275 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:26:12,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:26:12,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:26:12,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:26:12,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:26:12,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.69 MB 2025-02-15 15:26:12,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32080.72 MB 2025-02-15 15:26:12,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:26:12,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37004.25 MB 2025-02-15 15:26:12,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41200.65 MB 2025-02-15 15:26:12,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 15:26:12,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32080.72 MB 2025-02-15 15:26:12,445 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:26:12,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:26:12,446 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:26:12,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:26:12,447 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:26:12,452 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:26:12,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:26:12,453 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:26:12,453 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:27:53,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:27:53,739 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:27:53,747 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:27:53,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:27:53,753 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:27:53,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:27:53,755 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:28:12,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:28:12,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:28:12,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.86 seconds 2025-02-15 15:28:12,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:12,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-15 15:28:12,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-15 15:28:12,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-15 15:28:12,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53785.66 MB 2025-02-15 15:28:12,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33344.72 MB 2025-02-15 15:28:12,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20440.94 MB 2025-02-15 15:28:12,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-15 15:28:12,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:28:12,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:28:12,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 15:28:12,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:12,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-15 15:28:12,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-15 15:28:12,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-15 15:28:12,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33344.72 MB 2025-02-15 15:28:12,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41894.81 MB 2025-02-15 15:28:12,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8550.09 MB 2025-02-15 15:28:12,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38612.19 MB 2025-02-15 15:28:14,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:28:14,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:28:14,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:28:14,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:14,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-15 15:28:14,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-15 15:28:14,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:28:14,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41894.81 MB 2025-02-15 15:28:14,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29037.17 MB 2025-02-15 15:28:14,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12857.64 MB 2025-02-15 15:28:14,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26614.04 MB 2025-02-15 15:28:14,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:28:14,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:28:14,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:28:14,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:14,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 15:28:14,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-15 15:28:14,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:28:14,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 15:28:14,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29037.17 MB 2025-02-15 15:28:14,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:28:14,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-15 15:28:14,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:28:14,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:28:14,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:28:14,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:14,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-15 15:28:14,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 15:28:14,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:28:14,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 15:28:14,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-15 15:28:14,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5427.43 MB 2025-02-15 15:28:14,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32312.22 MB 2025-02-15 15:28:14,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:28:14,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:28:14,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:28:14,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:14,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-15 15:28:14,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-15 15:28:14,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:28:14,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29037.17 MB 2025-02-15 15:28:14,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-15 15:28:14,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5427.43 MB 2025-02-15 15:28:14,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32312.22 MB 2025-02-15 15:28:15,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:28:15,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:28:15,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:28:15,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:15,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28301.48 MB 2025-02-15 15:28:15,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29068.48 MB 2025-02-15 15:28:15,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:28:15,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-15 15:28:15,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34881.93 MB 2025-02-15 15:28:15,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:28:15,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29776.27 MB 2025-02-15 15:28:15,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:28:15,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:28:15,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:28:15,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:15,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29481.37 MB 2025-02-15 15:28:15,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29710.73 MB 2025-02-15 15:28:15,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.37 MB 2025-02-15 15:28:15,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34881.93 MB 2025-02-15 15:28:15,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34881.93 MB 2025-02-15 15:28:15,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:28:15,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29938.65 MB 2025-02-15 15:28:15,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:28:15,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:28:15,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.29 seconds 2025-02-15 15:28:15,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:15,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-15 15:28:15,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29911.24 MB 2025-02-15 15:28:15,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12702.41 MB 2025-02-15 15:28:15,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53785.66 MB 2025-02-15 15:28:15,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34881.93 MB 2025-02-15 15:28:15,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18903.73 MB 2025-02-15 15:28:15,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29938.65 MB 2025-02-15 15:28:15,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:28:15,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:28:15,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:28:15,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:15,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29911.24 MB 2025-02-15 15:28:15,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22205.85 MB 2025-02-15 15:28:15,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7705.39 MB 2025-02-15 15:28:15,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34881.93 MB 2025-02-15 15:28:15,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34881.93 MB 2025-02-15 15:28:15,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:28:15,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32416.19 MB 2025-02-15 15:28:15,336 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-15 15:28:15,336 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:28:15,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:28:15,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:28:15,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:28:15,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:28:15,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22205.85 MB 2025-02-15 15:28:15,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30620.80 MB 2025-02-15 15:28:15,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-15 15:28:15,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34881.93 MB 2025-02-15 15:28:15,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43249.57 MB 2025-02-15 15:28:15,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 15:28:15,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30620.80 MB 2025-02-15 15:28:15,501 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-15 15:28:15,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:28:15,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:28:15,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:28:15,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:28:15,508 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:28:15,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:28:15,509 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:28:15,509 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:29:40,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:29:40,835 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:29:40,844 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:29:40,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:29:40,851 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1941, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:29:40,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:29:40,853 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1941, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:30:11,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:30:11,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:30:11,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.25 seconds 2025-02-15 15:30:11,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:11,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26493.90 MB 2025-02-15 15:30:11,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33363.00 MB 2025-02-15 15:30:11,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6869.09 MB 2025-02-15 15:30:11,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51617.20 MB 2025-02-15 15:30:11,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40074.48 MB 2025-02-15 15:30:11,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11542.72 MB 2025-02-15 15:30:11,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42307.37 MB 2025-02-15 15:30:11,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:30:11,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:30:11,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-15 15:30:11,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:11,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33363.00 MB 2025-02-15 15:30:11,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25868.50 MB 2025-02-15 15:30:11,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7494.49 MB 2025-02-15 15:30:11,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40074.48 MB 2025-02-15 15:30:11,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54234.45 MB 2025-02-15 15:30:11,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14159.97 MB 2025-02-15 15:30:11,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52527.01 MB 2025-02-15 15:30:13,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:30:13,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:30:13,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:30:13,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25868.50 MB 2025-02-15 15:30:13,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26399.35 MB 2025-02-15 15:30:13,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:30:13,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54234.45 MB 2025-02-15 15:30:13,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34619.79 MB 2025-02-15 15:30:13,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19614.66 MB 2025-02-15 15:30:13,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30377.89 MB 2025-02-15 15:30:13,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:30:13,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:30:13,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:30:13,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-15 15:30:13,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28288.88 MB 2025-02-15 15:30:13,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:30:13,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34619.79 MB 2025-02-15 15:30:13,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34619.79 MB 2025-02-15 15:30:13,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:30:13,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.31 MB 2025-02-15 15:30:13,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:30:13,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:30:13,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:30:13,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28288.88 MB 2025-02-15 15:30:13,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-15 15:30:13,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:30:13,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34619.79 MB 2025-02-15 15:30:13,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39338.38 MB 2025-02-15 15:30:13,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 15:30:13,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-15 15:30:13,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:30:13,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:30:13,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:30:13,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-15 15:30:13,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-15 15:30:13,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:30:13,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34619.79 MB 2025-02-15 15:30:13,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39338.38 MB 2025-02-15 15:30:13,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-15 15:30:13,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-15 15:30:13,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:30:13,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:30:13,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 15:30:13,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32064.28 MB 2025-02-15 15:30:13,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32831.28 MB 2025-02-15 15:30:13,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:30:13,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39338.38 MB 2025-02-15 15:30:13,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 15:30:13,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 15:30:13,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.07 MB 2025-02-15 15:30:13,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:30:13,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:30:13,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 15:30:13,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33244.17 MB 2025-02-15 15:30:13,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33473.77 MB 2025-02-15 15:30:13,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.60 MB 2025-02-15 15:30:13,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 15:30:13,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 15:30:13,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:30:13,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33680.33 MB 2025-02-15 15:30:13,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:30:13,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:30:13,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.03 seconds 2025-02-15 15:30:13,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:13,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19731.31 MB 2025-02-15 15:30:13,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33674.84 MB 2025-02-15 15:30:13,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13943.54 MB 2025-02-15 15:30:13,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51617.20 MB 2025-02-15 15:30:13,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 15:30:13,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11865.69 MB 2025-02-15 15:30:13,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33680.33 MB 2025-02-15 15:30:14,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:30:14,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:30:14,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 15:30:14,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:14,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33674.84 MB 2025-02-15 15:30:14,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24735.69 MB 2025-02-15 15:30:14,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8939.15 MB 2025-02-15 15:30:14,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 15:30:14,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39751.52 MB 2025-02-15 15:30:14,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:30:14,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36186.51 MB 2025-02-15 15:30:14,188 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:30:14,188 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:30:14,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:30:14,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:30:14,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:30:14,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:30:14,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24735.69 MB 2025-02-15 15:30:14,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33174.72 MB 2025-02-15 15:30:14,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:30:14,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39751.52 MB 2025-02-15 15:30:14,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48142.22 MB 2025-02-15 15:30:14,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 15:30:14,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33174.72 MB 2025-02-15 15:30:14,355 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:30:14,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:14,356 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:30:14,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:14,357 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:30:14,362 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:30:14,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:14,363 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:30:14,363 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:30:52,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:52,609 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:30:52,613 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:30:52,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:52,617 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1757, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:30:52,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:30:52,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1757, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:31:20,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:31:20,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:31:20,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.43 seconds 2025-02-15 15:31:20,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:20,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25211.76 MB 2025-02-15 15:31:20,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31429.82 MB 2025-02-15 15:31:20,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6218.06 MB 2025-02-15 15:31:20,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60727.23 MB 2025-02-15 15:31:20,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39432.75 MB 2025-02-15 15:31:20,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21294.48 MB 2025-02-15 15:31:20,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40345.44 MB 2025-02-15 15:31:20,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:31:20,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:31:20,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 15:31:20,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:20,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.82 MB 2025-02-15 15:31:20,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24911.95 MB 2025-02-15 15:31:20,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6517.87 MB 2025-02-15 15:31:20,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39432.75 MB 2025-02-15 15:31:20,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51799.65 MB 2025-02-15 15:31:20,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12366.91 MB 2025-02-15 15:31:20,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47899.41 MB 2025-02-15 15:31:22,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:31:22,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:31:22,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:31:22,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24911.95 MB 2025-02-15 15:31:22,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25442.79 MB 2025-02-15 15:31:22,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:31:22,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51799.65 MB 2025-02-15 15:31:22,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30459.04 MB 2025-02-15 15:31:22,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21340.62 MB 2025-02-15 15:31:22,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29421.33 MB 2025-02-15 15:31:22,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:31:22,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:31:22,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:31:22,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-15 15:31:22,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27332.32 MB 2025-02-15 15:31:22,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:31:22,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30459.04 MB 2025-02-15 15:31:22,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30459.04 MB 2025-02-15 15:31:22,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:31:22,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28749.75 MB 2025-02-15 15:31:22,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:31:22,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:31:22,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:31:22,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27332.32 MB 2025-02-15 15:31:22,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-15 15:31:22,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:31:22,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30459.04 MB 2025-02-15 15:31:22,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37065.06 MB 2025-02-15 15:31:22,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 15:31:22,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-15 15:31:22,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:31:22,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:31:22,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:31:22,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-15 15:31:22,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-15 15:31:22,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:31:22,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30459.04 MB 2025-02-15 15:31:22,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37065.06 MB 2025-02-15 15:31:22,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 15:31:22,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-15 15:31:22,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:31:22,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:31:22,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:31:22,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31107.72 MB 2025-02-15 15:31:22,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31874.72 MB 2025-02-15 15:31:22,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:31:22,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37065.06 MB 2025-02-15 15:31:22,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37478.20 MB 2025-02-15 15:31:22,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 15:31:22,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32582.51 MB 2025-02-15 15:31:22,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:31:22,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:31:22,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:31:22,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32287.61 MB 2025-02-15 15:31:22,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32515.75 MB 2025-02-15 15:31:22,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.14 MB 2025-02-15 15:31:22,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37478.20 MB 2025-02-15 15:31:22,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37478.20 MB 2025-02-15 15:31:22,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:31:22,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32729.99 MB 2025-02-15 15:31:22,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:31:22,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:31:22,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.93 seconds 2025-02-15 15:31:22,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19090.23 MB 2025-02-15 15:31:22,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32715.62 MB 2025-02-15 15:31:22,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13625.38 MB 2025-02-15 15:31:22,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60727.23 MB 2025-02-15 15:31:22,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37478.20 MB 2025-02-15 15:31:22,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23249.03 MB 2025-02-15 15:31:22,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32729.99 MB 2025-02-15 15:31:22,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:31:22,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:31:22,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:31:22,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32715.62 MB 2025-02-15 15:31:22,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24076.93 MB 2025-02-15 15:31:22,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8638.69 MB 2025-02-15 15:31:22,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37478.20 MB 2025-02-15 15:31:22,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37478.20 MB 2025-02-15 15:31:22,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:31:22,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35212.23 MB 2025-02-15 15:31:22,837 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-15 15:31:22,837 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:31:22,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:31:22,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:31:22,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:31:22,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:22,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24076.93 MB 2025-02-15 15:31:22,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32465.35 MB 2025-02-15 15:31:22,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-15 15:31:22,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37478.20 MB 2025-02-15 15:31:22,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45818.58 MB 2025-02-15 15:31:22,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-15 15:31:22,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32465.35 MB 2025-02-15 15:31:23,001 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-15 15:31:23,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:23,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:31:23,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:23,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:31:23,008 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:31:23,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:23,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:31:23,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:31:35,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:35,811 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:31:35,816 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:31:35,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:35,819 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:31:35,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:35,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:31:48,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:31:48,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:31:48,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.79 seconds 2025-02-15 15:31:48,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:48,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-15 15:31:48,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-15 15:31:48,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-15 15:31:48,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58328.09 MB 2025-02-15 15:31:48,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24943.53 MB 2025-02-15 15:31:48,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33384.56 MB 2025-02-15 15:31:48,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.08 MB 2025-02-15 15:31:48,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:31:48,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:31:48,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 15:31:48,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:48,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-15 15:31:48,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19978.39 MB 2025-02-15 15:31:48,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1481.10 MB 2025-02-15 15:31:48,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24943.53 MB 2025-02-15 15:31:48,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31769.76 MB 2025-02-15 15:31:48,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6826.23 MB 2025-02-15 15:31:48,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31093.62 MB 2025-02-15 15:31:50,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:31:50,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:31:50,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:31:50,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:50,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.39 MB 2025-02-15 15:31:50,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20509.23 MB 2025-02-15 15:31:50,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:31:50,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31769.76 MB 2025-02-15 15:31:50,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23498.59 MB 2025-02-15 15:31:50,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8271.17 MB 2025-02-15 15:31:50,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24489.86 MB 2025-02-15 15:31:50,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:31:50,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:31:50,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:31:50,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:50,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-15 15:31:50,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22398.77 MB 2025-02-15 15:31:50,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:31:50,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23498.59 MB 2025-02-15 15:31:50,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25386.02 MB 2025-02-15 15:31:50,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:31:50,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23816.20 MB 2025-02-15 15:31:50,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:31:50,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:31:50,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:31:50,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:50,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.77 MB 2025-02-15 15:31:50,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-15 15:31:50,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:31:50,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25386.02 MB 2025-02-15 15:31:50,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31992.05 MB 2025-02-15 15:31:50,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 15:31:50,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-15 15:31:50,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:31:50,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:31:50,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:31:50,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:50,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-15 15:31:50,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-15 15:31:50,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:31:50,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23498.59 MB 2025-02-15 15:31:50,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31992.05 MB 2025-02-15 15:31:50,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 15:31:50,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-15 15:31:51,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:31:51,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:31:51,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:31:51,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:51,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26174.16 MB 2025-02-15 15:31:51,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26941.17 MB 2025-02-15 15:31:51,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:31:51,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31992.05 MB 2025-02-15 15:31:51,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 15:31:51,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-15 15:31:51,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27648.95 MB 2025-02-15 15:31:51,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:31:51,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:31:51,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:31:51,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:51,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.06 MB 2025-02-15 15:31:51,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27582.91 MB 2025-02-15 15:31:51,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.85 MB 2025-02-15 15:31:51,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 15:31:51,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 15:31:51,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:31:51,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27793.84 MB 2025-02-15 15:31:51,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:31:51,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:31:51,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.21 seconds 2025-02-15 15:31:51,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:51,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-15 15:31:51,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27783.24 MB 2025-02-15 15:31:51,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11999.40 MB 2025-02-15 15:31:51,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58328.09 MB 2025-02-15 15:31:51,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 15:31:51,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25924.99 MB 2025-02-15 15:31:51,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27793.84 MB 2025-02-15 15:31:51,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:31:51,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:31:51,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:31:51,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:51,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27783.24 MB 2025-02-15 15:31:51,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20777.32 MB 2025-02-15 15:31:51,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7005.92 MB 2025-02-15 15:31:51,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 15:31:51,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-15 15:31:51,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:31:51,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30286.21 MB 2025-02-15 15:31:51,326 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-15 15:31:51,326 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:31:51,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:31:51,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:31:51,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:31:51,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:31:51,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20777.32 MB 2025-02-15 15:31:51,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29186.62 MB 2025-02-15 15:31:51,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 15:31:51,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-15 15:31:51,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40762.34 MB 2025-02-15 15:31:51,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 15:31:51,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29186.62 MB 2025-02-15 15:31:51,489 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-15 15:31:51,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:51,490 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:31:51,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:51,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:31:51,496 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:31:51,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:31:51,497 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:31:51,497 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:32:00,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:00,320 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:32:00,325 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:32:00,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:00,328 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 237, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:32:00,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:00,329 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 237, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:32:04,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:32:04,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:32:04,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.71 seconds 2025-02-15 15:32:04,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:04,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14620.16 MB 2025-02-15 15:32:04,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15458.89 MB 2025-02-15 15:32:04,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 838.73 MB 2025-02-15 15:32:04,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49121.59 MB 2025-02-15 15:32:04,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18368.95 MB 2025-02-15 15:32:04,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30752.64 MB 2025-02-15 15:32:04,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24318.02 MB 2025-02-15 15:32:04,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:32:04,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:32:04,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:32:04,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:04,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15458.89 MB 2025-02-15 15:32:04,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15773.89 MB 2025-02-15 15:32:04,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.00 MB 2025-02-15 15:32:04,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18368.95 MB 2025-02-15 15:32:04,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20340.28 MB 2025-02-15 15:32:04,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1971.32 MB 2025-02-15 15:32:04,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18605.20 MB 2025-02-15 15:32:05,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:32:05,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:32:05,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.08 seconds 2025-02-15 15:32:05,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15773.89 MB 2025-02-15 15:32:05,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16071.16 MB 2025-02-15 15:32:05,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 297.27 MB 2025-02-15 15:32:05,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20340.28 MB 2025-02-15 15:32:05,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19547.55 MB 2025-02-15 15:32:05,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -792.72 MB 2025-02-15 15:32:05,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20029.51 MB 2025-02-15 15:32:05,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:32:05,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:32:05,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:32:05,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16071.16 MB 2025-02-15 15:32:05,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17129.04 MB 2025-02-15 15:32:05,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1057.88 MB 2025-02-15 15:32:05,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19547.55 MB 2025-02-15 15:32:05,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19547.55 MB 2025-02-15 15:32:05,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:32:05,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17922.80 MB 2025-02-15 15:32:05,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:32:05,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:32:05,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 15:32:05,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17129.04 MB 2025-02-15 15:32:05,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18384.51 MB 2025-02-15 15:32:05,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1255.47 MB 2025-02-15 15:32:05,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19547.55 MB 2025-02-15 15:32:05,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22718.45 MB 2025-02-15 15:32:05,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3170.89 MB 2025-02-15 15:32:05,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21489.28 MB 2025-02-15 15:32:05,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:32:05,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:32:05,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 15:32:05,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16071.16 MB 2025-02-15 15:32:05,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18384.51 MB 2025-02-15 15:32:05,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2313.35 MB 2025-02-15 15:32:05,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19547.55 MB 2025-02-15 15:32:05,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22718.45 MB 2025-02-15 15:32:05,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3170.89 MB 2025-02-15 15:32:05,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21489.28 MB 2025-02-15 15:32:05,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:32:05,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:32:05,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 15:32:05,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19243.29 MB 2025-02-15 15:32:05,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19672.81 MB 2025-02-15 15:32:05,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 429.52 MB 2025-02-15 15:32:05,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22718.45 MB 2025-02-15 15:32:05,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22947.04 MB 2025-02-15 15:32:05,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 228.59 MB 2025-02-15 15:32:05,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20069.17 MB 2025-02-15 15:32:05,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:32:05,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:32:05,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:32:05,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19904.04 MB 2025-02-15 15:32:05,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20132.32 MB 2025-02-15 15:32:05,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.29 MB 2025-02-15 15:32:05,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22947.04 MB 2025-02-15 15:32:05,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22947.04 MB 2025-02-15 15:32:05,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:32:05,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20181.11 MB 2025-02-15 15:32:05,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:32:05,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:32:05,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.13 seconds 2025-02-15 15:32:05,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13794.43 MB 2025-02-15 15:32:05,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20332.95 MB 2025-02-15 15:32:05,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6538.52 MB 2025-02-15 15:32:05,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49121.59 MB 2025-02-15 15:32:05,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22947.04 MB 2025-02-15 15:32:05,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26174.55 MB 2025-02-15 15:32:05,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20332.95 MB 2025-02-15 15:32:05,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:32:05,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:32:05,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:32:05,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14953.68 MB 2025-02-15 15:32:05,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17961.08 MB 2025-02-15 15:32:05,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.40 MB 2025-02-15 15:32:05,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22947.04 MB 2025-02-15 15:32:05,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22947.04 MB 2025-02-15 15:32:05,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:32:05,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18261.78 MB 2025-02-15 15:32:05,749 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 15:32:05,750 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:32:05,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:32:05,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:32:05,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:32:05,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:32:05,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17961.08 MB 2025-02-15 15:32:05,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26381.85 MB 2025-02-15 15:32:05,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 15:32:05,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22947.04 MB 2025-02-15 15:32:05,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31318.87 MB 2025-02-15 15:32:05,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-15 15:32:05,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26381.85 MB 2025-02-15 15:32:05,914 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 15:32:05,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:05,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:32:05,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:05,917 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:32:05,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:32:05,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:32:05,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:32:05,922 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:33:26,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:26,257 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:33:26,263 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:33:26,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:26,267 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 168, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:33:26,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:26,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 168, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:33:28,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:33:28,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:33:28,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.59 seconds 2025-02-15 15:33:28,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:28,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.36 MB 2025-02-15 15:33:28,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14733.90 MB 2025-02-15 15:33:28,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.54 MB 2025-02-15 15:33:28,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39690.70 MB 2025-02-15 15:33:28,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18752.73 MB 2025-02-15 15:33:28,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20937.97 MB 2025-02-15 15:33:28,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23610.73 MB 2025-02-15 15:33:28,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:33:28,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:33:28,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:33:28,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:28,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14733.90 MB 2025-02-15 15:33:28,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14979.82 MB 2025-02-15 15:33:28,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.92 MB 2025-02-15 15:33:28,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18752.73 MB 2025-02-15 15:33:28,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18752.73 MB 2025-02-15 15:33:28,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:33:28,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17009.42 MB 2025-02-15 15:33:29,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:33:29,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:33:29,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 15:33:29,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14979.82 MB 2025-02-15 15:33:29,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15194.81 MB 2025-02-15 15:33:29,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-15 15:33:29,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18752.73 MB 2025-02-15 15:33:29,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18683.53 MB 2025-02-15 15:33:29,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -69.21 MB 2025-02-15 15:33:29,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19150.50 MB 2025-02-15 15:33:29,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:33:29,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:33:29,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:33:29,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-15 15:33:29,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15959.82 MB 2025-02-15 15:33:29,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-15 15:33:29,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18683.53 MB 2025-02-15 15:33:29,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18683.53 MB 2025-02-15 15:33:29,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:33:29,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16533.88 MB 2025-02-15 15:33:29,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:33:29,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:33:29,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 15:33:29,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15959.82 MB 2025-02-15 15:33:29,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-15 15:33:29,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-15 15:33:29,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18683.53 MB 2025-02-15 15:33:29,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20602.42 MB 2025-02-15 15:33:29,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1918.89 MB 2025-02-15 15:33:29,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-15 15:33:29,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:33:29,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:33:29,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 15:33:29,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-15 15:33:29,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-15 15:33:29,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-15 15:33:29,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18683.53 MB 2025-02-15 15:33:29,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20602.42 MB 2025-02-15 15:33:29,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1918.89 MB 2025-02-15 15:33:29,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-15 15:33:29,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:33:29,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:33:29,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:33:29,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17488.89 MB 2025-02-15 15:33:29,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17800.51 MB 2025-02-15 15:33:29,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.62 MB 2025-02-15 15:33:29,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20602.42 MB 2025-02-15 15:33:29,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20768.10 MB 2025-02-15 15:33:29,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 15:33:29,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18093.86 MB 2025-02-15 15:33:29,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:33:29,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:33:29,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:33:29,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17967.74 MB 2025-02-15 15:33:29,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18172.88 MB 2025-02-15 15:33:29,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.14 MB 2025-02-15 15:33:29,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20768.10 MB 2025-02-15 15:33:29,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20772.29 MB 2025-02-15 15:33:29,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-15 15:33:29,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18195.55 MB 2025-02-15 15:33:29,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:33:29,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:33:29,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.62 seconds 2025-02-15 15:33:29,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:29,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13554.03 MB 2025-02-15 15:33:29,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18373.78 MB 2025-02-15 15:33:29,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4819.75 MB 2025-02-15 15:33:29,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39690.70 MB 2025-02-15 15:33:29,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20772.29 MB 2025-02-15 15:33:29,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18918.41 MB 2025-02-15 15:33:29,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18373.78 MB 2025-02-15 15:33:30,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:33:30,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:33:30,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:33:30,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:30,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.78 MB 2025-02-15 15:33:30,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17433.55 MB 2025-02-15 15:33:30,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -940.24 MB 2025-02-15 15:33:30,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20772.29 MB 2025-02-15 15:33:30,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20772.29 MB 2025-02-15 15:33:30,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:33:30,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.45 MB 2025-02-15 15:33:30,181 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-15 15:33:30,181 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:33:30,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:33:30,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:33:30,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:33:30,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:33:30,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17433.55 MB 2025-02-15 15:33:30,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25865.01 MB 2025-02-15 15:33:30,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-15 15:33:30,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20772.29 MB 2025-02-15 15:33:30,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31253.86 MB 2025-02-15 15:33:30,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-15 15:33:30,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25865.01 MB 2025-02-15 15:33:30,351 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-15 15:33:30,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:30,352 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:33:30,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:30,353 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:33:30,358 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:33:30,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:33:30,359 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:33:30,359 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:34:57,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:34:57,794 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:34:57,799 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:34:57,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:34:57,805 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2057, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:34:57,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:34:57,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2057, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:35:29,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:35:29,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:35:29,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.82 seconds 2025-02-15 15:35:29,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:29,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27302.21 MB 2025-02-15 15:35:29,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34581.82 MB 2025-02-15 15:35:29,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7279.61 MB 2025-02-15 15:35:29,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39638.27 MB 2025-02-15 15:35:29,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40527.46 MB 2025-02-15 15:35:29,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 889.19 MB 2025-02-15 15:35:29,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43568.35 MB 2025-02-15 15:35:29,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:35:29,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:35:29,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 15:35:29,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:29,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34581.82 MB 2025-02-15 15:35:29,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26472.60 MB 2025-02-15 15:35:29,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8109.22 MB 2025-02-15 15:35:29,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40527.46 MB 2025-02-15 15:35:29,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56990.11 MB 2025-02-15 15:35:29,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16462.64 MB 2025-02-15 15:35:29,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56139.49 MB 2025-02-15 15:35:31,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:35:31,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:35:31,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:35:31,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:31,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26472.60 MB 2025-02-15 15:35:31,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27003.44 MB 2025-02-15 15:35:31,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:35:31,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56990.11 MB 2025-02-15 15:35:31,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31178.36 MB 2025-02-15 15:35:31,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25811.75 MB 2025-02-15 15:35:31,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30982.14 MB 2025-02-15 15:35:31,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:35:31,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:35:31,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:35:31,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:31,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27003.44 MB 2025-02-15 15:35:31,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.98 MB 2025-02-15 15:35:31,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:35:31,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31178.36 MB 2025-02-15 15:35:31,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32122.08 MB 2025-02-15 15:35:31,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 15:35:31,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30310.41 MB 2025-02-15 15:35:31,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:35:31,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:35:31,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:35:31,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:31,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28892.98 MB 2025-02-15 15:35:31,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31134.83 MB 2025-02-15 15:35:31,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:35:31,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32122.08 MB 2025-02-15 15:35:31,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38728.11 MB 2025-02-15 15:35:31,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 15:35:31,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36679.11 MB 2025-02-15 15:35:31,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:35:31,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:35:31,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:35:31,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:31,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27003.44 MB 2025-02-15 15:35:31,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31134.83 MB 2025-02-15 15:35:31,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:35:31,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31178.36 MB 2025-02-15 15:35:31,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38728.11 MB 2025-02-15 15:35:31,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:35:31,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36679.11 MB 2025-02-15 15:35:32,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:35:32,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:35:32,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:35:32,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:32,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32668.37 MB 2025-02-15 15:35:32,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33435.38 MB 2025-02-15 15:35:32,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:35:32,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38728.11 MB 2025-02-15 15:35:32,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39145.44 MB 2025-02-15 15:35:32,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:35:32,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34143.16 MB 2025-02-15 15:35:32,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:35:32,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:35:32,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:35:32,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:32,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33848.27 MB 2025-02-15 15:35:32,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34077.99 MB 2025-02-15 15:35:32,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.72 MB 2025-02-15 15:35:32,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39145.44 MB 2025-02-15 15:35:32,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39145.44 MB 2025-02-15 15:35:32,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:35:32,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34323.74 MB 2025-02-15 15:35:32,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:35:32,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:35:32,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.37 seconds 2025-02-15 15:35:32,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:32,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20135.46 MB 2025-02-15 15:35:32,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34278.67 MB 2025-02-15 15:35:32,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14143.21 MB 2025-02-15 15:35:32,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39638.27 MB 2025-02-15 15:35:32,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39145.44 MB 2025-02-15 15:35:32,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -492.83 MB 2025-02-15 15:35:32,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34323.74 MB 2025-02-15 15:35:32,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:35:32,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:35:32,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:35:32,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:32,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34278.67 MB 2025-02-15 15:35:32,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25133.93 MB 2025-02-15 15:35:32,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9144.74 MB 2025-02-15 15:35:32,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39145.44 MB 2025-02-15 15:35:32,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39145.44 MB 2025-02-15 15:35:32,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:35:32,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36785.42 MB 2025-02-15 15:35:32,463 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 15:35:32,463 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:35:32,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:35:32,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:35:32,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:35:32,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:35:32,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25133.93 MB 2025-02-15 15:35:32,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33556.25 MB 2025-02-15 15:35:32,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 15:35:32,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39145.44 MB 2025-02-15 15:35:32,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47519.37 MB 2025-02-15 15:35:32,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 15:35:32,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33556.25 MB 2025-02-15 15:35:32,635 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 15:35:32,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:35:32,636 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:35:32,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:35:32,637 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:35:32,642 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:35:32,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:35:32,643 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:35:32,643 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:36:28,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:36:28,321 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:36:28,326 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:36:28,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:36:28,330 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:36:28,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:36:28,331 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:37:01,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:37:01,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:37:01,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.64 seconds 2025-02-15 15:37:01,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:01,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27978.12 MB 2025-02-15 15:37:01,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35601.27 MB 2025-02-15 15:37:01,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7623.15 MB 2025-02-15 15:37:01,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60079.21 MB 2025-02-15 15:37:01,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40846.23 MB 2025-02-15 15:37:01,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19232.98 MB 2025-02-15 15:37:01,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44470.76 MB 2025-02-15 15:37:02,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:37:02,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:37:02,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 15:37:02,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:02,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35601.27 MB 2025-02-15 15:37:02,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26976.87 MB 2025-02-15 15:37:02,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8624.40 MB 2025-02-15 15:37:02,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40846.23 MB 2025-02-15 15:37:02,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57321.46 MB 2025-02-15 15:37:02,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16475.23 MB 2025-02-15 15:37:02,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57183.88 MB 2025-02-15 15:37:04,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:37:04,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:37:04,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 15:37:04,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26976.87 MB 2025-02-15 15:37:04,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27507.72 MB 2025-02-15 15:37:04,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:37:04,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57321.46 MB 2025-02-15 15:37:04,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31161.58 MB 2025-02-15 15:37:04,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26159.87 MB 2025-02-15 15:37:04,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31487.30 MB 2025-02-15 15:37:04,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:37:04,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:37:04,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:37:04,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-15 15:37:04,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29397.25 MB 2025-02-15 15:37:04,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:37:04,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31161.58 MB 2025-02-15 15:37:04,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-15 15:37:04,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:37:04,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30814.68 MB 2025-02-15 15:37:04,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:37:04,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:37:04,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:37:04,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29397.25 MB 2025-02-15 15:37:04,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31639.11 MB 2025-02-15 15:37:04,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:37:04,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33049.02 MB 2025-02-15 15:37:04,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38711.33 MB 2025-02-15 15:37:04,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:37:04,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.39 MB 2025-02-15 15:37:04,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:37:04,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:37:04,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:37:04,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-15 15:37:04,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31639.11 MB 2025-02-15 15:37:04,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:37:04,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31161.58 MB 2025-02-15 15:37:04,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38711.33 MB 2025-02-15 15:37:04,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:37:04,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.39 MB 2025-02-15 15:37:04,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:37:04,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:37:04,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:37:04,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33172.65 MB 2025-02-15 15:37:04,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33939.65 MB 2025-02-15 15:37:04,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:37:04,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38711.33 MB 2025-02-15 15:37:04,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 15:37:04,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:37:04,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34647.44 MB 2025-02-15 15:37:04,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:37:04,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:37:04,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:37:04,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34352.54 MB 2025-02-15 15:37:04,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34581.73 MB 2025-02-15 15:37:04,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.19 MB 2025-02-15 15:37:04,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 15:37:04,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 15:37:04,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:37:04,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34812.34 MB 2025-02-15 15:37:04,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:37:04,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:37:04,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.22 seconds 2025-02-15 15:37:04,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-15 15:37:04,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34782.81 MB 2025-02-15 15:37:04,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14309.39 MB 2025-02-15 15:37:04,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60079.21 MB 2025-02-15 15:37:04,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 15:37:04,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20950.55 MB 2025-02-15 15:37:04,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34812.34 MB 2025-02-15 15:37:04,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:37:04,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:37:04,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:37:04,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34782.81 MB 2025-02-15 15:37:04,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25477.80 MB 2025-02-15 15:37:04,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9305.00 MB 2025-02-15 15:37:04,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 15:37:04,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39128.66 MB 2025-02-15 15:37:04,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:37:04,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37294.47 MB 2025-02-15 15:37:04,862 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:37:04,863 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:37:04,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:37:04,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:37:04,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 15:37:04,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:04,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25477.80 MB 2025-02-15 15:37:04,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33916.83 MB 2025-02-15 15:37:04,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:37:04,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39128.66 MB 2025-02-15 15:37:04,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47519.37 MB 2025-02-15 15:37:04,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 15:37:04,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.83 MB 2025-02-15 15:37:05,033 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:37:05,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:05,034 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:37:05,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:05,035 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:37:05,040 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:37:05,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:05,041 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:37:05,041 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:37:19,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:19,709 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:37:19,714 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:37:19,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:19,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:37:19,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:19,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:37:39,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:37:39,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:37:39,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.87 seconds 2025-02-15 15:37:39,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:39,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-15 15:37:39,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-15 15:37:39,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-15 15:37:39,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60104.38 MB 2025-02-15 15:37:39,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37719.38 MB 2025-02-15 15:37:39,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22385.00 MB 2025-02-15 15:37:39,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-15 15:37:39,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:37:39,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:37:39,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 15:37:39,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:39,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-15 15:37:39,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-15 15:37:39,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-15 15:37:39,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-15 15:37:39,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46607.11 MB 2025-02-15 15:37:39,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8887.73 MB 2025-02-15 15:37:39,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39643.54 MB 2025-02-15 15:37:41,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:37:41,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:37:41,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:37:41,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:41,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-15 15:37:41,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-15 15:37:41,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:37:41,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46607.11 MB 2025-02-15 15:37:41,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33233.57 MB 2025-02-15 15:37:41,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13373.54 MB 2025-02-15 15:37:41,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26873.98 MB 2025-02-15 15:37:41,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:37:41,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:37:41,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:37:41,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:41,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 15:37:41,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-15 15:37:41,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:37:41,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33233.57 MB 2025-02-15 15:37:41,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33233.57 MB 2025-02-15 15:37:41,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:37:41,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-15 15:37:41,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:37:41,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:37:41,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:37:41,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:41,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-15 15:37:41,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 15:37:41,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:37:41,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33233.57 MB 2025-02-15 15:37:41,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-15 15:37:41,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 15:37:41,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 15:37:41,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:37:41,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:37:41,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:37:41,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:41,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-15 15:37:41,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-15 15:37:41,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:37:41,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33233.57 MB 2025-02-15 15:37:41,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-15 15:37:41,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-15 15:37:41,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-15 15:37:42,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:37:42,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:37:42,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:37:42,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:42,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-15 15:37:42,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-15 15:37:42,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:37:42,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-15 15:37:42,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-15 15:37:42,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:37:42,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-15 15:37:42,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:37:42,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:37:42,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:37:42,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:42,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-15 15:37:42,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29966.36 MB 2025-02-15 15:37:42,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.11 MB 2025-02-15 15:37:42,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-15 15:37:42,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-15 15:37:42,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:37:42,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30207.77 MB 2025-02-15 15:37:42,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:37:42,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:37:42,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.31 seconds 2025-02-15 15:37:42,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:42,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-15 15:37:42,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30166.26 MB 2025-02-15 15:37:42,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.22 MB 2025-02-15 15:37:42,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60104.38 MB 2025-02-15 15:37:42,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-15 15:37:42,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25983.71 MB 2025-02-15 15:37:42,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30207.77 MB 2025-02-15 15:37:42,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:37:42,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:37:42,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:37:42,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:42,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30166.26 MB 2025-02-15 15:37:42,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22370.10 MB 2025-02-15 15:37:42,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7796.16 MB 2025-02-15 15:37:42,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-15 15:37:42,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-15 15:37:42,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:37:42,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32663.18 MB 2025-02-15 15:37:42,313 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 15:37:42,314 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:37:42,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:37:42,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:37:42,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:37:42,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:37:42,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22370.10 MB 2025-02-15 15:37:42,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30759.24 MB 2025-02-15 15:37:42,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 15:37:42,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-15 15:37:42,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38291.90 MB 2025-02-15 15:37:42,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-15 15:37:42,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30759.24 MB 2025-02-15 15:37:42,477 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 15:37:42,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:42,478 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:37:42,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:42,479 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:37:42,484 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:37:42,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:37:42,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:37:42,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:38:43,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:43,670 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:38:43,675 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:38:43,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:43,678 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:38:43,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:43,679 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:38:48,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:38:48,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:38:48,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.78 seconds 2025-02-15 15:38:48,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:48,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15094.00 MB 2025-02-15 15:38:48,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16174.03 MB 2025-02-15 15:38:48,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1080.03 MB 2025-02-15 15:38:48,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46634.37 MB 2025-02-15 15:38:48,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17559.45 MB 2025-02-15 15:38:48,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29074.92 MB 2025-02-15 15:38:48,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25019.16 MB 2025-02-15 15:38:48,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:38:48,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:38:48,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 15:38:48,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:48,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16174.03 MB 2025-02-15 15:38:48,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16612.92 MB 2025-02-15 15:38:48,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 438.89 MB 2025-02-15 15:38:48,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17559.45 MB 2025-02-15 15:38:48,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21162.36 MB 2025-02-15 15:38:48,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3602.91 MB 2025-02-15 15:38:48,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20291.39 MB 2025-02-15 15:38:49,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:38:49,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:38:49,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.48 seconds 2025-02-15 15:38:49,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:49,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16612.92 MB 2025-02-15 15:38:49,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17001.76 MB 2025-02-15 15:38:49,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 388.84 MB 2025-02-15 15:38:49,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21162.36 MB 2025-02-15 15:38:49,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18020.83 MB 2025-02-15 15:38:49,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3141.53 MB 2025-02-15 15:38:49,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20953.48 MB 2025-02-15 15:38:49,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:38:49,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:38:49,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:38:49,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:49,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-15 15:38:49,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18387.98 MB 2025-02-15 15:38:49,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1386.22 MB 2025-02-15 15:38:49,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18020.83 MB 2025-02-15 15:38:49,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20789.07 MB 2025-02-15 15:38:49,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2768.24 MB 2025-02-15 15:38:49,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19427.43 MB 2025-02-15 15:38:50,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:38:50,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:38:50,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:38:50,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18387.98 MB 2025-02-15 15:38:50,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20031.73 MB 2025-02-15 15:38:50,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1643.75 MB 2025-02-15 15:38:50,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20789.07 MB 2025-02-15 15:38:50,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25633.49 MB 2025-02-15 15:38:50,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4844.42 MB 2025-02-15 15:38:50,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24097.62 MB 2025-02-15 15:38:50,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:38:50,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:38:50,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:38:50,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-15 15:38:50,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20031.73 MB 2025-02-15 15:38:50,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3029.97 MB 2025-02-15 15:38:50,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18020.83 MB 2025-02-15 15:38:50,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25633.49 MB 2025-02-15 15:38:50,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7612.66 MB 2025-02-15 15:38:50,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24097.62 MB 2025-02-15 15:38:50,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:38:50,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:38:50,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:38:50,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21155.45 MB 2025-02-15 15:38:50,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21718.45 MB 2025-02-15 15:38:50,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 563.01 MB 2025-02-15 15:38:50,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25633.49 MB 2025-02-15 15:38:50,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25935.48 MB 2025-02-15 15:38:50,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 301.99 MB 2025-02-15 15:38:50,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22236.91 MB 2025-02-15 15:38:50,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:38:50,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:38:50,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:38:50,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22020.90 MB 2025-02-15 15:38:50,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22239.76 MB 2025-02-15 15:38:50,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.87 MB 2025-02-15 15:38:50,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25935.48 MB 2025-02-15 15:38:50,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25935.48 MB 2025-02-15 15:38:50,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:38:50,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22326.87 MB 2025-02-15 15:38:50,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:38:50,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:38:50,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.76 seconds 2025-02-15 15:38:50,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14031.35 MB 2025-02-15 15:38:50,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22440.84 MB 2025-02-15 15:38:50,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.49 MB 2025-02-15 15:38:50,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46634.37 MB 2025-02-15 15:38:50,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25935.48 MB 2025-02-15 15:38:50,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20698.89 MB 2025-02-15 15:38:50,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22440.84 MB 2025-02-15 15:38:50,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:38:50,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:38:50,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 15:38:50,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22440.84 MB 2025-02-15 15:38:50,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25454.87 MB 2025-02-15 15:38:50,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 15:38:50,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25935.48 MB 2025-02-15 15:38:50,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27009.22 MB 2025-02-15 15:38:50,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-15 15:38:50,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25756.50 MB 2025-02-15 15:38:50,743 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:38:50,743 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:38:50,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:38:50,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:38:50,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:38:50,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:38:50,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18532.35 MB 2025-02-15 15:38:50,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26971.37 MB 2025-02-15 15:38:50,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:38:50,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27009.22 MB 2025-02-15 15:38:50,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-15 15:38:50,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:38:50,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26971.37 MB 2025-02-15 15:38:50,910 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:38:50,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:50,912 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:38:50,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:50,913 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:38:50,917 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:38:50,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:38:50,919 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:38:50,919 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:40:07,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:07,510 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:40:07,515 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:40:07,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:07,519 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1260, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:40:07,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:07,520 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1260, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:40:26,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:40:26,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:40:26,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.46 seconds 2025-02-15 15:40:26,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:26,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21748.59 MB 2025-02-15 15:40:26,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26207.66 MB 2025-02-15 15:40:26,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4459.07 MB 2025-02-15 15:40:26,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50084.18 MB 2025-02-15 15:40:26,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37717.28 MB 2025-02-15 15:40:26,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12366.91 MB 2025-02-15 15:40:26,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35070.33 MB 2025-02-15 15:40:27,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:40:27,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:40:27,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:40:27,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:27,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26207.66 MB 2025-02-15 15:40:27,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22328.20 MB 2025-02-15 15:40:27,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3879.46 MB 2025-02-15 15:40:27,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37717.28 MB 2025-02-15 15:40:27,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46571.45 MB 2025-02-15 15:40:27,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8854.18 MB 2025-02-15 15:40:27,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39515.78 MB 2025-02-15 15:40:28,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:40:28,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:40:28,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:40:28,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:28,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22328.20 MB 2025-02-15 15:40:28,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22859.04 MB 2025-02-15 15:40:28,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:40:28,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46571.45 MB 2025-02-15 15:40:28,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29062.33 MB 2025-02-15 15:40:28,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17509.12 MB 2025-02-15 15:40:28,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26837.59 MB 2025-02-15 15:40:29,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:40:29,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:40:29,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:40:29,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-15 15:40:29,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24748.57 MB 2025-02-15 15:40:29,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:40:29,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 15:40:29,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29062.33 MB 2025-02-15 15:40:29,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:40:29,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.00 MB 2025-02-15 15:40:29,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:40:29,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:40:29,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:40:29,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24748.57 MB 2025-02-15 15:40:29,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-15 15:40:29,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:40:29,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 15:40:29,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 15:40:29,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:40:29,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-15 15:40:29,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:40:29,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:40:29,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:40:29,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-15 15:40:29,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-15 15:40:29,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:40:29,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29062.33 MB 2025-02-15 15:40:29,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34724.64 MB 2025-02-15 15:40:29,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:40:29,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-15 15:40:29,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:40:29,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:40:29,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 15:40:29,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28523.97 MB 2025-02-15 15:40:29,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.97 MB 2025-02-15 15:40:29,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:40:29,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34724.64 MB 2025-02-15 15:40:29,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 15:40:29,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:40:29,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29998.76 MB 2025-02-15 15:40:29,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:40:29,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:40:29,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:40:29,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29703.86 MB 2025-02-15 15:40:29,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29932.09 MB 2025-02-15 15:40:29,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-15 15:40:29,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 15:40:29,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 15:40:29,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:40:29,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30163.39 MB 2025-02-15 15:40:29,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:40:29,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:40:29,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.91 seconds 2025-02-15 15:40:29,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17358.65 MB 2025-02-15 15:40:29,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30132.94 MB 2025-02-15 15:40:29,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12774.29 MB 2025-02-15 15:40:29,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50084.18 MB 2025-02-15 15:40:29,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 15:40:29,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14942.21 MB 2025-02-15 15:40:29,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30163.39 MB 2025-02-15 15:40:29,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:40:29,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:40:29,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:40:29,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30132.94 MB 2025-02-15 15:40:29,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22349.27 MB 2025-02-15 15:40:29,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7783.66 MB 2025-02-15 15:40:29,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 15:40:29,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-15 15:40:29,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:40:29,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32632.93 MB 2025-02-15 15:40:29,724 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 15:40:29,725 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:40:29,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:40:29,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:40:29,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:40:29,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:40:29,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22349.27 MB 2025-02-15 15:40:29,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30749.18 MB 2025-02-15 15:40:29,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-15 15:40:29,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-15 15:40:29,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39317.41 MB 2025-02-15 15:40:29,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 15:40:29,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30749.18 MB 2025-02-15 15:40:29,893 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 15:40:29,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:29,895 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:40:29,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:29,897 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:40:29,902 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:40:29,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:40:29,903 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:40:29,903 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:41:47,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:41:47,240 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:41:47,245 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:41:47,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:41:47,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:41:47,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:41:47,250 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:42:18,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:42:18,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:42:18,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.84 seconds 2025-02-15 15:42:18,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:18,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.47 MB 2025-02-15 15:42:18,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33835.81 MB 2025-02-15 15:42:18,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7028.34 MB 2025-02-15 15:42:18,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47668.26 MB 2025-02-15 15:42:18,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40227.57 MB 2025-02-15 15:42:18,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7440.70 MB 2025-02-15 15:42:18,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42847.12 MB 2025-02-15 15:42:18,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:42:18,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:42:18,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:42:18,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:18,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33835.81 MB 2025-02-15 15:42:18,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.45 MB 2025-02-15 15:42:18,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7733.37 MB 2025-02-15 15:42:18,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40227.57 MB 2025-02-15 15:42:18,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54758.74 MB 2025-02-15 15:42:18,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14531.17 MB 2025-02-15 15:42:18,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53558.35 MB 2025-02-15 15:42:20,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:42:20,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:42:20,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-15 15:42:20,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.45 MB 2025-02-15 15:42:20,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26633.29 MB 2025-02-15 15:42:20,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:42:20,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54758.74 MB 2025-02-15 15:42:20,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30438.06 MB 2025-02-15 15:42:20,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24320.67 MB 2025-02-15 15:42:20,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30612.87 MB 2025-02-15 15:42:20,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:42:20,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:42:20,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:42:20,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-15 15:42:20,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28522.82 MB 2025-02-15 15:42:20,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:42:20,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30438.06 MB 2025-02-15 15:42:20,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32325.50 MB 2025-02-15 15:42:20,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:42:20,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.25 MB 2025-02-15 15:42:20,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:42:20,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:42:20,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:42:20,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28522.82 MB 2025-02-15 15:42:20,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-15 15:42:20,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:42:20,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32325.50 MB 2025-02-15 15:42:20,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-15 15:42:20,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:42:20,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-15 15:42:20,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:42:20,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:42:20,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:42:20,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-15 15:42:20,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-15 15:42:20,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:42:20,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30438.06 MB 2025-02-15 15:42:20,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-15 15:42:20,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:42:20,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-15 15:42:20,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:42:20,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:42:20,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:42:20,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32298.22 MB 2025-02-15 15:42:20,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33065.22 MB 2025-02-15 15:42:20,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:42:20,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37987.81 MB 2025-02-15 15:42:20,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 15:42:20,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:42:20,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.01 MB 2025-02-15 15:42:20,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:42:20,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:42:20,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:42:20,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33478.11 MB 2025-02-15 15:42:20,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33707.07 MB 2025-02-15 15:42:20,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 15:42:20,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-15 15:42:20,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 15:42:20,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:42:20,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.28 MB 2025-02-15 15:42:20,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:42:20,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:42:20,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.43 seconds 2025-02-15 15:42:20,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19888.09 MB 2025-02-15 15:42:20,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33907.95 MB 2025-02-15 15:42:20,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14019.86 MB 2025-02-15 15:42:20,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47668.26 MB 2025-02-15 15:42:20,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 15:42:20,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9263.12 MB 2025-02-15 15:42:20,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.28 MB 2025-02-15 15:42:20,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:42:20,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:42:20,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:42:20,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33907.95 MB 2025-02-15 15:42:20,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24889.43 MB 2025-02-15 15:42:20,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9018.52 MB 2025-02-15 15:42:20,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-15 15:42:20,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-15 15:42:20,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:42:20,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36417.16 MB 2025-02-15 15:42:20,969 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 15:42:20,969 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:42:20,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:42:20,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:42:20,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:42:20,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:20,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24889.43 MB 2025-02-15 15:42:20,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33319.86 MB 2025-02-15 15:42:20,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.43 MB 2025-02-15 15:42:20,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-15 15:42:20,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42597.35 MB 2025-02-15 15:42:20,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 15:42:20,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33319.86 MB 2025-02-15 15:42:21,136 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 15:42:21,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:21,138 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:42:21,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:21,139 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:42:21,145 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:42:21,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:21,146 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:42:21,146 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:42:27,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:27,432 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:42:27,437 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:42:27,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:27,441 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1931, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:42:27,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:42:27,441 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1931, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:42:57,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:42:57,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:42:57,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.42 seconds 2025-02-15 15:42:57,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:57,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26424.22 MB 2025-02-15 15:42:57,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33258.84 MB 2025-02-15 15:42:57,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6834.62 MB 2025-02-15 15:42:57,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50977.57 MB 2025-02-15 15:42:57,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40047.21 MB 2025-02-15 15:42:57,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10930.36 MB 2025-02-15 15:42:57,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42237.38 MB 2025-02-15 15:42:58,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:42:58,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:42:58,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 15:42:58,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:58,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33258.84 MB 2025-02-15 15:42:58,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25816.52 MB 2025-02-15 15:42:58,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7442.32 MB 2025-02-15 15:42:58,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40047.21 MB 2025-02-15 15:42:58,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54431.58 MB 2025-02-15 15:42:58,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14384.37 MB 2025-02-15 15:42:58,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52758.35 MB 2025-02-15 15:42:59,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:42:59,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:42:59,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:42:59,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:59,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25816.52 MB 2025-02-15 15:42:59,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26347.36 MB 2025-02-15 15:42:59,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:42:59,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54431.58 MB 2025-02-15 15:42:59,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30452.74 MB 2025-02-15 15:42:59,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23978.84 MB 2025-02-15 15:42:59,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30325.91 MB 2025-02-15 15:42:59,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:42:59,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:42:59,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:42:59,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:42:59,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-15 15:42:59,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28236.89 MB 2025-02-15 15:42:59,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:42:59,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30452.74 MB 2025-02-15 15:42:59,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32340.18 MB 2025-02-15 15:42:59,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:42:59,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29654.32 MB 2025-02-15 15:43:00,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:43:00,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:43:00,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 15:43:00,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28236.89 MB 2025-02-15 15:43:00,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-15 15:43:00,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:43:00,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32340.18 MB 2025-02-15 15:43:00,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38002.49 MB 2025-02-15 15:43:00,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:43:00,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-15 15:43:00,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:43:00,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:43:00,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:43:00,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-15 15:43:00,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-15 15:43:00,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:43:00,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30452.74 MB 2025-02-15 15:43:00,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38002.49 MB 2025-02-15 15:43:00,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:43:00,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-15 15:43:00,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:43:00,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:43:00,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:43:00,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32012.29 MB 2025-02-15 15:43:00,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32779.29 MB 2025-02-15 15:43:00,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:43:00,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38002.49 MB 2025-02-15 15:43:00,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38415.63 MB 2025-02-15 15:43:00,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 15:43:00,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.08 MB 2025-02-15 15:43:00,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:43:00,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:43:00,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:43:00,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33192.18 MB 2025-02-15 15:43:00,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33420.02 MB 2025-02-15 15:43:00,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.84 MB 2025-02-15 15:43:00,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38415.63 MB 2025-02-15 15:43:00,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38415.63 MB 2025-02-15 15:43:00,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:00,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.03 MB 2025-02-15 15:43:00,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:43:00,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:43:00,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.93 seconds 2025-02-15 15:43:00,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19696.46 MB 2025-02-15 15:43:00,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33619.94 MB 2025-02-15 15:43:00,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13923.48 MB 2025-02-15 15:43:00,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50977.57 MB 2025-02-15 15:43:00,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38415.63 MB 2025-02-15 15:43:00,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12561.94 MB 2025-02-15 15:43:00,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.03 MB 2025-02-15 15:43:00,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:43:00,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:43:00,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:43:00,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33619.94 MB 2025-02-15 15:43:00,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24683.88 MB 2025-02-15 15:43:00,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8936.06 MB 2025-02-15 15:43:00,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38415.63 MB 2025-02-15 15:43:00,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38415.63 MB 2025-02-15 15:43:00,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:00,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36117.17 MB 2025-02-15 15:43:00,658 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-15 15:43:00,658 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:43:00,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:43:00,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:43:00,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:43:00,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:00,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24683.88 MB 2025-02-15 15:43:00,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33074.93 MB 2025-02-15 15:43:00,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-15 15:43:00,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38415.63 MB 2025-02-15 15:43:00,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46758.10 MB 2025-02-15 15:43:00,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 15:43:00,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33074.93 MB 2025-02-15 15:43:00,823 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-15 15:43:00,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:00,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:43:00,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:00,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:43:00,832 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:43:00,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:00,833 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:43:00,833 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:43:55,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:55,488 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:43:55,493 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:43:55,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:55,498 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 106, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:43:55,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:55,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 106, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:43:57,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:43:57,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:43:57,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-15 15:43:57,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-15 15:43:57,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.46 MB 2025-02-15 15:43:57,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 375.13 MB 2025-02-15 15:43:57,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55100.57 MB 2025-02-15 15:43:57,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 15:43:57,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37679.53 MB 2025-02-15 15:43:57,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22952.21 MB 2025-02-15 15:43:57,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:43:57,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:43:57,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:43:57,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.46 MB 2025-02-15 15:43:57,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14264.21 MB 2025-02-15 15:43:57,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 181.75 MB 2025-02-15 15:43:57,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 15:43:57,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 15:43:57,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:57,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14826.96 MB 2025-02-15 15:43:57,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:43:57,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:43:57,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.55 seconds 2025-02-15 15:43:57,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.21 MB 2025-02-15 15:43:57,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14404.88 MB 2025-02-15 15:43:57,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 140.67 MB 2025-02-15 15:43:57,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 15:43:57,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 15:43:57,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:57,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18349.96 MB 2025-02-15 15:43:57,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:43:57,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:43:57,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:43:57,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-15 15:43:57,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14905.42 MB 2025-02-15 15:43:57,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 500.60 MB 2025-02-15 15:43:57,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 15:43:57,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17421.04 MB 2025-02-15 15:43:57,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:57,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15281.05 MB 2025-02-15 15:43:57,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:43:57,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:43:57,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 15:43:57,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14905.42 MB 2025-02-15 15:43:57,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-15 15:43:57,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.03 MB 2025-02-15 15:43:57,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 15:43:57,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17924.36 MB 2025-02-15 15:43:57,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 503.32 MB 2025-02-15 15:43:57,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-15 15:43:57,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:43:57,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:43:57,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 15:43:57,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-15 15:43:57,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-15 15:43:57,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1108.64 MB 2025-02-15 15:43:57,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17421.04 MB 2025-02-15 15:43:57,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17924.36 MB 2025-02-15 15:43:57,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 503.32 MB 2025-02-15 15:43:57,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-15 15:43:57,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:43:57,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:43:57,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 15:43:57,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16100.46 MB 2025-02-15 15:43:57,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16355.82 MB 2025-02-15 15:43:57,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.36 MB 2025-02-15 15:43:57,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17924.36 MB 2025-02-15 15:43:57,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18087.94 MB 2025-02-15 15:43:57,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-15 15:43:57,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16543.38 MB 2025-02-15 15:43:57,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:43:57,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:43:57,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:43:57,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16517.34 MB 2025-02-15 15:43:57,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16745.07 MB 2025-02-15 15:43:57,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-15 15:43:57,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18087.94 MB 2025-02-15 15:43:57,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18087.94 MB 2025-02-15 15:43:57,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:43:57,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16745.07 MB 2025-02-15 15:43:57,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:43:57,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:43:57,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-15 15:43:57,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:57,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13338.02 MB 2025-02-15 15:43:57,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16946.07 MB 2025-02-15 15:43:57,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3608.05 MB 2025-02-15 15:43:57,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55100.57 MB 2025-02-15 15:43:57,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18087.94 MB 2025-02-15 15:43:57,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37012.64 MB 2025-02-15 15:43:57,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16946.07 MB 2025-02-15 15:43:58,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:43:58,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:43:58,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 15:43:58,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:58,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14037.55 MB 2025-02-15 15:43:58,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17050.48 MB 2025-02-15 15:43:58,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-15 15:43:58,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18087.94 MB 2025-02-15 15:43:58,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18490.59 MB 2025-02-15 15:43:58,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 15:43:58,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17352.05 MB 2025-02-15 15:43:58,187 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-15 15:43:58,187 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:43:58,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:43:58,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:43:58,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:43:58,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:43:58,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17050.48 MB 2025-02-15 15:43:58,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25486.07 MB 2025-02-15 15:43:58,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-15 15:43:58,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18490.59 MB 2025-02-15 15:43:58,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28976.35 MB 2025-02-15 15:43:58,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-15 15:43:58,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25486.07 MB 2025-02-15 15:43:58,353 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-15 15:43:58,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:58,354 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:43:58,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:58,355 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:43:58,360 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:43:58,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:43:58,362 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:43:58,362 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:45:28,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:28,409 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:45:28,414 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:45:28,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:28,418 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:45:28,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:28,419 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:45:46,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:45:46,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:45:46,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.65 seconds 2025-02-15 15:45:46,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:46,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20926.35 MB 2025-02-15 15:45:46,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24967.82 MB 2025-02-15 15:45:46,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-15 15:45:46,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37364.96 MB 2025-02-15 15:45:46,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26814.19 MB 2025-02-15 15:45:46,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10550.77 MB 2025-02-15 15:45:46,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33795.91 MB 2025-02-15 15:45:46,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:45:46,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:45:46,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 15:45:46,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:46,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24967.82 MB 2025-02-15 15:45:46,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.80 MB 2025-02-15 15:45:46,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3252.02 MB 2025-02-15 15:45:46,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26814.19 MB 2025-02-15 15:45:46,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36926.65 MB 2025-02-15 15:45:46,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10112.47 MB 2025-02-15 15:45:46,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37180.33 MB 2025-02-15 15:45:48,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:45:48,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:45:48,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 15:45:48,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.80 MB 2025-02-15 15:45:48,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22246.64 MB 2025-02-15 15:45:48,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:45:48,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36926.65 MB 2025-02-15 15:45:48,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24895.29 MB 2025-02-15 15:45:48,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12031.36 MB 2025-02-15 15:45:48,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26226.23 MB 2025-02-15 15:45:48,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:45:48,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:45:48,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:45:48,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22246.64 MB 2025-02-15 15:45:48,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24136.18 MB 2025-02-15 15:45:48,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:45:48,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 15:45:48,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27726.45 MB 2025-02-15 15:45:48,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 15:45:48,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25553.61 MB 2025-02-15 15:45:48,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:45:48,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:45:48,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:45:48,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24136.18 MB 2025-02-15 15:45:48,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26378.03 MB 2025-02-15 15:45:48,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:45:48,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27726.45 MB 2025-02-15 15:45:48,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 15:45:48,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:45:48,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31922.31 MB 2025-02-15 15:45:48,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:45:48,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:45:48,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:45:48,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22246.64 MB 2025-02-15 15:45:48,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26378.03 MB 2025-02-15 15:45:48,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:45:48,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24895.29 MB 2025-02-15 15:45:48,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33388.76 MB 2025-02-15 15:45:48,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-15 15:45:48,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31922.31 MB 2025-02-15 15:45:48,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:45:48,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:45:48,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:45:48,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27911.58 MB 2025-02-15 15:45:48,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28678.58 MB 2025-02-15 15:45:48,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:45:48,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33388.76 MB 2025-02-15 15:45:48,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 15:45:48,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:45:48,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29386.37 MB 2025-02-15 15:45:48,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:45:48,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:45:48,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:45:48,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29091.47 MB 2025-02-15 15:45:48,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29319.17 MB 2025-02-15 15:45:48,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.71 MB 2025-02-15 15:45:48,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 15:45:48,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 15:45:48,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:45:48,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.07 MB 2025-02-15 15:45:48,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:45:48,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:45:48,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.14 seconds 2025-02-15 15:45:48,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16947.53 MB 2025-02-15 15:45:48,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29520.02 MB 2025-02-15 15:45:48,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12572.50 MB 2025-02-15 15:45:48,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37364.96 MB 2025-02-15 15:45:48,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 15:45:48,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3560.96 MB 2025-02-15 15:45:48,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.07 MB 2025-02-15 15:45:48,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:45:48,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:45:48,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:45:48,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29520.02 MB 2025-02-15 15:45:48,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21946.35 MB 2025-02-15 15:45:48,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7573.68 MB 2025-02-15 15:45:48,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 15:45:48,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33803.99 MB 2025-02-15 15:45:48,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:45:48,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32027.08 MB 2025-02-15 15:45:48,854 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-15 15:45:48,854 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:45:48,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:45:48,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:45:48,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:45:48,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:45:48,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21946.35 MB 2025-02-15 15:45:48,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30369.56 MB 2025-02-15 15:45:48,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-15 15:45:48,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33803.99 MB 2025-02-15 15:45:48,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42180.02 MB 2025-02-15 15:45:48,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 15:45:48,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30369.56 MB 2025-02-15 15:45:49,022 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-15 15:45:49,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:49,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:45:49,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:49,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:45:49,029 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:45:49,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:45:49,030 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:45:49,030 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:47:10,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:10,094 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:47:10,103 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:47:10,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:10,113 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:47:10,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:10,115 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:47:43,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:47:43,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:47:43,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.54 seconds 2025-02-15 15:47:43,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:43,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27964.19 MB 2025-02-15 15:47:43,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.04 MB 2025-02-15 15:47:43,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7616.86 MB 2025-02-15 15:47:43,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-15 15:47:43,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40848.33 MB 2025-02-15 15:47:43,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9707.72 MB 2025-02-15 15:47:43,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44456.82 MB 2025-02-15 15:47:43,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:47:43,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:47:43,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 15:47:43,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:43,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35581.04 MB 2025-02-15 15:47:43,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26966.48 MB 2025-02-15 15:47:43,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8614.57 MB 2025-02-15 15:47:43,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40848.33 MB 2025-02-15 15:47:43,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56396.61 MB 2025-02-15 15:47:43,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15548.28 MB 2025-02-15 15:47:43,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55768.22 MB 2025-02-15 15:47:45,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:47:45,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:47:45,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:47:45,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:45,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26966.48 MB 2025-02-15 15:47:45,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27497.32 MB 2025-02-15 15:47:45,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:47:45,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56396.61 MB 2025-02-15 15:47:45,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31167.87 MB 2025-02-15 15:47:45,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25228.74 MB 2025-02-15 15:47:45,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31476.90 MB 2025-02-15 15:47:45,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:47:45,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:47:45,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:47:45,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:45,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-15 15:47:45,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29386.85 MB 2025-02-15 15:47:45,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:47:45,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31167.87 MB 2025-02-15 15:47:45,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33055.31 MB 2025-02-15 15:47:45,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:47:45,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30804.28 MB 2025-02-15 15:47:46,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:47:46,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:47:46,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:47:46,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29386.85 MB 2025-02-15 15:47:46,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-15 15:47:46,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:47:46,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33055.31 MB 2025-02-15 15:47:46,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38717.62 MB 2025-02-15 15:47:46,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:47:46,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-15 15:47:46,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:47:46,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:47:46,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:47:46,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-15 15:47:46,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-15 15:47:46,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:47:46,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31167.87 MB 2025-02-15 15:47:46,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38717.62 MB 2025-02-15 15:47:46,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:47:46,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-15 15:47:46,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:47:46,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:47:46,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:47:46,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33162.25 MB 2025-02-15 15:47:46,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33929.25 MB 2025-02-15 15:47:46,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:47:46,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38717.62 MB 2025-02-15 15:47:46,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39132.86 MB 2025-02-15 15:47:46,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:47:46,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34637.04 MB 2025-02-15 15:47:46,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:47:46,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:47:46,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:47:46,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34342.14 MB 2025-02-15 15:47:46,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34570.12 MB 2025-02-15 15:47:46,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-15 15:47:46,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39132.86 MB 2025-02-15 15:47:46,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39132.86 MB 2025-02-15 15:47:46,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:47:46,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.94 MB 2025-02-15 15:47:46,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:47:46,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:47:46,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.09 seconds 2025-02-15 15:47:46,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20466.45 MB 2025-02-15 15:47:46,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34770.97 MB 2025-02-15 15:47:46,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14304.52 MB 2025-02-15 15:47:46,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-15 15:47:46,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39132.86 MB 2025-02-15 15:47:46,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11423.19 MB 2025-02-15 15:47:46,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.94 MB 2025-02-15 15:47:46,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:47:46,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:47:46,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:47:46,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34770.97 MB 2025-02-15 15:47:46,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25453.51 MB 2025-02-15 15:47:46,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9317.46 MB 2025-02-15 15:47:46,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39132.86 MB 2025-02-15 15:47:46,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39132.86 MB 2025-02-15 15:47:46,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:47:46,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37267.89 MB 2025-02-15 15:47:46,501 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-15 15:47:46,501 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:47:46,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:47:46,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:47:46,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:47:46,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:47:46,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25453.51 MB 2025-02-15 15:47:46,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33842.65 MB 2025-02-15 15:47:46,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-15 15:47:46,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39132.86 MB 2025-02-15 15:47:46,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47475.33 MB 2025-02-15 15:47:46,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-15 15:47:46,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33842.65 MB 2025-02-15 15:47:46,673 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-15 15:47:46,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:46,675 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:47:46,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:46,676 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:47:46,681 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:47:46,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:47:46,682 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:47:46,682 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:48:38,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:48:38,866 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:48:38,873 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:48:38,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:48:38,879 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:48:38,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:48:38,881 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:49:09,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:49:09,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:49:09,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.82 seconds 2025-02-15 15:49:09,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:09,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-15 15:49:09,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-15 15:49:09,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-15 15:49:09,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55817.80 MB 2025-02-15 15:49:09,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40099.64 MB 2025-02-15 15:49:09,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15718.15 MB 2025-02-15 15:49:09,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42439.46 MB 2025-02-15 15:49:09,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:49:09,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:49:09,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:49:09,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:09,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-15 15:49:09,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-15 15:49:09,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-15 15:49:09,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40099.64 MB 2025-02-15 15:49:09,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54454.65 MB 2025-02-15 15:49:09,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14355.01 MB 2025-02-15 15:49:09,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53021.68 MB 2025-02-15 15:49:11,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:49:11,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:49:11,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-15 15:49:11,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:11,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-15 15:49:11,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-15 15:49:11,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:49:11,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54454.65 MB 2025-02-15 15:49:11,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-15 15:49:11,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24048.04 MB 2025-02-15 15:49:11,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30477.71 MB 2025-02-15 15:49:11,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:49:11,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:49:11,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:49:11,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:11,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-15 15:49:11,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-15 15:49:11,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:49:11,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 15:49:11,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32294.04 MB 2025-02-15 15:49:11,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:49:11,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-15 15:49:12,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:49:12,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:49:12,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:49:12,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-15 15:49:12,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-15 15:49:12,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:49:12,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32294.04 MB 2025-02-15 15:49:12,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37956.35 MB 2025-02-15 15:49:12,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:49:12,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-15 15:49:12,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:49:12,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:49:12,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:49:12,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-15 15:49:12,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-15 15:49:12,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:49:12,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30406.61 MB 2025-02-15 15:49:12,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37956.35 MB 2025-02-15 15:49:12,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-15 15:49:12,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-15 15:49:12,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:49:12,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:49:12,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:49:12,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-15 15:49:12,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-15 15:49:12,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:49:12,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37956.35 MB 2025-02-15 15:49:12,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-15 15:49:12,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:49:12,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-15 15:49:12,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:49:12,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:49:12,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:49:12,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-15 15:49:12,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33571.86 MB 2025-02-15 15:49:12,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-15 15:49:12,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-15 15:49:12,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-15 15:49:12,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:49:12,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33759.38 MB 2025-02-15 15:49:12,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:49:12,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:49:12,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.40 seconds 2025-02-15 15:49:12,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-15 15:49:12,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33772.71 MB 2025-02-15 15:49:12,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13975.20 MB 2025-02-15 15:49:12,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55817.80 MB 2025-02-15 15:49:12,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-15 15:49:12,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17444.11 MB 2025-02-15 15:49:12,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.71 MB 2025-02-15 15:49:12,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:49:12,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:49:12,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:49:12,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33772.71 MB 2025-02-15 15:49:12,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24798.11 MB 2025-02-15 15:49:12,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8974.60 MB 2025-02-15 15:49:12,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-15 15:49:12,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-15 15:49:12,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:49:12,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36281.30 MB 2025-02-15 15:49:12,575 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-15 15:49:12,576 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:49:12,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:49:12,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:49:12,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:49:12,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:12,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24798.11 MB 2025-02-15 15:49:12,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33227.23 MB 2025-02-15 15:49:12,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-15 15:49:12,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-15 15:49:12,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46753.91 MB 2025-02-15 15:49:12,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-15 15:49:12,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33227.23 MB 2025-02-15 15:49:12,742 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-15 15:49:12,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:12,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:49:12,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:12,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:49:12,751 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:49:12,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:12,752 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:49:12,752 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:49:20,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:20,472 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:49:20,477 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:49:20,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:20,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:49:20,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:20,481 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:49:38,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:49:38,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:49:38,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.79 seconds 2025-02-15 15:49:38,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:38,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20884.54 MB 2025-02-15 15:49:38,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.78 MB 2025-02-15 15:49:38,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-15 15:49:38,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55134.13 MB 2025-02-15 15:49:38,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28840.03 MB 2025-02-15 15:49:38,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26294.09 MB 2025-02-15 15:49:38,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33753.29 MB 2025-02-15 15:49:38,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:49:38,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:49:38,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 15:49:38,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:38,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.78 MB 2025-02-15 15:49:38,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21684.61 MB 2025-02-15 15:49:38,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3220.17 MB 2025-02-15 15:49:38,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28840.03 MB 2025-02-15 15:49:38,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38944.11 MB 2025-02-15 15:49:38,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10104.08 MB 2025-02-15 15:49:38,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37110.27 MB 2025-02-15 15:49:40,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:49:40,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:49:40,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:49:40,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21684.61 MB 2025-02-15 15:49:40,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22215.45 MB 2025-02-15 15:49:40,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:49:40,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38944.11 MB 2025-02-15 15:49:40,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26944.21 MB 2025-02-15 15:49:40,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11999.90 MB 2025-02-15 15:49:40,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26194.00 MB 2025-02-15 15:49:40,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:49:40,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:49:40,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:49:40,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-15 15:49:40,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24104.98 MB 2025-02-15 15:49:40,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:49:40,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 15:49:40,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-15 15:49:40,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 15:49:40,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25522.41 MB 2025-02-15 15:49:40,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:49:40,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:49:40,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:49:40,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24104.98 MB 2025-02-15 15:49:40,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26346.84 MB 2025-02-15 15:49:40,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:49:40,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-15 15:49:40,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 15:49:40,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 15:49:40,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.12 MB 2025-02-15 15:49:40,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:49:40,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:49:40,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:49:40,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-15 15:49:40,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26346.84 MB 2025-02-15 15:49:40,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:49:40,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26944.21 MB 2025-02-15 15:49:40,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33550.24 MB 2025-02-15 15:49:40,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 15:49:40,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.12 MB 2025-02-15 15:49:40,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:49:40,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:49:40,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:49:40,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27880.38 MB 2025-02-15 15:49:40,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28647.38 MB 2025-02-15 15:49:40,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:49:40,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33550.24 MB 2025-02-15 15:49:40,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 15:49:40,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:49:40,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29355.17 MB 2025-02-15 15:49:40,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:49:40,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:49:40,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:49:40,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29060.27 MB 2025-02-15 15:49:40,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29289.23 MB 2025-02-15 15:49:40,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 15:49:40,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 15:49:40,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 15:49:40,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:49:40,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29524.51 MB 2025-02-15 15:49:40,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:49:40,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:49:40,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.23 seconds 2025-02-15 15:49:40,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16926.62 MB 2025-02-15 15:49:40,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29490.09 MB 2025-02-15 15:49:40,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12563.46 MB 2025-02-15 15:49:40,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55134.13 MB 2025-02-15 15:49:40,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 15:49:40,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21166.56 MB 2025-02-15 15:49:40,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29524.51 MB 2025-02-15 15:49:40,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:49:40,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:49:40,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:49:40,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:40,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29490.09 MB 2025-02-15 15:49:40,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21918.67 MB 2025-02-15 15:49:40,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7571.41 MB 2025-02-15 15:49:40,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 15:49:40,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33967.57 MB 2025-02-15 15:49:40,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:49:40,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31991.31 MB 2025-02-15 15:49:40,997 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-15 15:49:40,997 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:49:41,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:49:41,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:49:41,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:49:41,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:49:41,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21918.67 MB 2025-02-15 15:49:41,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30323.75 MB 2025-02-15 15:49:41,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-15 15:49:41,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33967.57 MB 2025-02-15 15:49:41,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42322.62 MB 2025-02-15 15:49:41,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-15 15:49:41,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30323.75 MB 2025-02-15 15:49:41,161 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-15 15:49:41,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:41,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:49:41,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:41,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:49:41,168 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:49:41,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:49:41,169 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:49:41,169 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:51:28,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:28,709 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:51:28,714 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:51:28,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:28,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:51:28,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:28,719 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:51:31,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:51:31,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:51:31,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.68 seconds 2025-02-15 15:51:31,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:31,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-15 15:51:31,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-15 15:51:31,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-15 15:51:31,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50677.68 MB 2025-02-15 15:51:31,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18366.86 MB 2025-02-15 15:51:31,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32310.82 MB 2025-02-15 15:51:31,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23652.54 MB 2025-02-15 15:51:31,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:51:31,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:51:31,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:51:31,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:31,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-15 15:51:31,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15004.64 MB 2025-02-15 15:51:31,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.70 MB 2025-02-15 15:51:31,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18366.86 MB 2025-02-15 15:51:31,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18937.28 MB 2025-02-15 15:51:31,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-15 15:51:31,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17059.08 MB 2025-02-15 15:51:32,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:51:32,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:51:32,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-15 15:51:32,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15004.64 MB 2025-02-15 15:51:32,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15218.30 MB 2025-02-15 15:51:32,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-15 15:51:32,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18937.28 MB 2025-02-15 15:51:32,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 15:51:32,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-15 15:51:32,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19175.40 MB 2025-02-15 15:51:32,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:51:32,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:51:32,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:51:32,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15218.24 MB 2025-02-15 15:51:32,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15978.59 MB 2025-02-15 15:51:32,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-15 15:51:32,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 15:51:32,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-15 15:51:32,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:51:32,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16549.11 MB 2025-02-15 15:51:32,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:51:32,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:51:32,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 15:51:32,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15978.59 MB 2025-02-15 15:51:32,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.98 MB 2025-02-15 15:51:32,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-15 15:51:32,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 15:51:32,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20755.51 MB 2025-02-15 15:51:32,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-15 15:51:32,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.51 MB 2025-02-15 15:51:32,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:51:32,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:51:32,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 15:51:32,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15218.24 MB 2025-02-15 15:51:32,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.98 MB 2025-02-15 15:51:32,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-15 15:51:32,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-15 15:51:32,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20755.51 MB 2025-02-15 15:51:32,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-15 15:51:32,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.51 MB 2025-02-15 15:51:32,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:51:32,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:51:32,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:51:32,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.23 MB 2025-02-15 15:51:32,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17806.95 MB 2025-02-15 15:51:32,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-15 15:51:32,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20755.51 MB 2025-02-15 15:51:32,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-15 15:51:32,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-15 15:51:32,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18100.21 MB 2025-02-15 15:51:32,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:51:32,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:51:32,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:51:32,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17973.14 MB 2025-02-15 15:51:32,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18200.86 MB 2025-02-15 15:51:32,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-15 15:51:32,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-15 15:51:32,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-15 15:51:32,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:51:32,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18224.52 MB 2025-02-15 15:51:32,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:51:32,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:51:32,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.70 seconds 2025-02-15 15:51:32,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-15 15:51:32,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18401.94 MB 2025-02-15 15:51:32,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4827.00 MB 2025-02-15 15:51:32,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50677.68 MB 2025-02-15 15:51:32,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-15 15:51:32,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29756.49 MB 2025-02-15 15:51:32,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18401.94 MB 2025-02-15 15:51:32,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:51:32,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:51:32,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:51:32,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18401.94 MB 2025-02-15 15:51:32,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17451.15 MB 2025-02-15 15:51:32,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -950.79 MB 2025-02-15 15:51:32,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-15 15:51:32,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-15 15:51:32,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:51:32,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19105.20 MB 2025-02-15 15:51:32,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:51:32,706 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 15:51:32,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:51:32,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:51:32,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:51:32,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:51:32,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17451.15 MB 2025-02-15 15:51:32,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25890.17 MB 2025-02-15 15:51:32,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:51:32,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-15 15:51:32,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31411.14 MB 2025-02-15 15:51:32,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:51:32,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25890.17 MB 2025-02-15 15:51:32,877 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:51:32,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:32,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:51:32,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:32,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:51:32,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:51:32,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:32,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:51:32,886 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-15 15:51:54,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:54,040 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:51:54,045 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:51:54,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:54,049 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2564, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:51:54,050 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:51:54,050 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2564, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:52:33,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:52:33,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:52:33,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.85 seconds 2025-02-15 15:52:33,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:33,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30835.07 MB 2025-02-15 15:52:33,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39909.45 MB 2025-02-15 15:52:33,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9074.38 MB 2025-02-15 15:52:33,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61868.08 MB 2025-02-15 15:52:33,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43423.63 MB 2025-02-15 15:52:33,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18444.45 MB 2025-02-15 15:52:33,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48983.30 MB 2025-02-15 15:52:34,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:52:34,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:52:34,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-15 15:52:34,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:34,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39909.45 MB 2025-02-15 15:52:34,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29108.34 MB 2025-02-15 15:52:34,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10801.11 MB 2025-02-15 15:52:34,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43423.63 MB 2025-02-15 15:52:34,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63235.42 MB 2025-02-15 15:52:34,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19811.79 MB 2025-02-15 15:52:34,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66509.45 MB 2025-02-15 15:52:36,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:52:36,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:52:36,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:52:36,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29108.34 MB 2025-02-15 15:52:36,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.18 MB 2025-02-15 15:52:36,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:52:36,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63235.42 MB 2025-02-15 15:52:36,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31935.43 MB 2025-02-15 15:52:36,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31299.99 MB 2025-02-15 15:52:36,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.72 MB 2025-02-15 15:52:36,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:52:36,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:52:36,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:52:36,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29639.18 MB 2025-02-15 15:52:36,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31528.71 MB 2025-02-15 15:52:36,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:52:36,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31935.43 MB 2025-02-15 15:52:36,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34766.59 MB 2025-02-15 15:52:36,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 15:52:36,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32946.14 MB 2025-02-15 15:52:36,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:52:36,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:52:36,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:52:36,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31528.71 MB 2025-02-15 15:52:36,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33770.57 MB 2025-02-15 15:52:36,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:52:36,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34766.59 MB 2025-02-15 15:52:36,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40900.76 MB 2025-02-15 15:52:36,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:52:36,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39314.85 MB 2025-02-15 15:52:36,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:52:36,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:52:36,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:52:36,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29639.18 MB 2025-02-15 15:52:36,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33770.57 MB 2025-02-15 15:52:36,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:52:36,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31935.43 MB 2025-02-15 15:52:36,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40900.76 MB 2025-02-15 15:52:36,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 15:52:36,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39314.85 MB 2025-02-15 15:52:36,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:52:36,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:52:36,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:52:36,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35304.11 MB 2025-02-15 15:52:36,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36071.11 MB 2025-02-15 15:52:36,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:52:36,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40900.76 MB 2025-02-15 15:52:36,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 15:52:36,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:52:36,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36778.90 MB 2025-02-15 15:52:36,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:52:36,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:52:36,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:52:36,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36484.00 MB 2025-02-15 15:52:36,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36711.77 MB 2025-02-15 15:52:36,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.77 MB 2025-02-15 15:52:36,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 15:52:36,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 15:52:36,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:52:36,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36917.64 MB 2025-02-15 15:52:36,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:52:36,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:52:36,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.47 seconds 2025-02-15 15:52:36,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21901.89 MB 2025-02-15 15:52:36,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36912.08 MB 2025-02-15 15:52:36,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15010.19 MB 2025-02-15 15:52:36,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52932.12 MB 2025-02-15 15:52:36,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 15:52:36,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11614.03 MB 2025-02-15 15:52:36,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36917.64 MB 2025-02-15 15:52:36,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:52:36,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:52:36,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:52:36,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36912.08 MB 2025-02-15 15:52:36,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26894.38 MB 2025-02-15 15:52:36,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10017.70 MB 2025-02-15 15:52:36,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 15:52:36,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41318.09 MB 2025-02-15 15:52:36,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:52:36,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39414.23 MB 2025-02-15 15:52:36,811 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-15 15:52:36,811 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:52:36,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:52:36,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:52:36,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:52:36,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:52:36,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26894.38 MB 2025-02-15 15:52:36,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35302.12 MB 2025-02-15 15:52:36,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-15 15:52:36,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41318.09 MB 2025-02-15 15:52:36,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45497.71 MB 2025-02-15 15:52:36,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-15 15:52:36,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35302.12 MB 2025-02-15 15:52:36,977 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-15 15:52:36,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:52:36,979 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:52:36,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:52:36,980 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:52:36,985 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:52:36,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:52:36,986 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:52:36,986 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:53:09,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:09,990 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:53:09,995 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:53:09,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:09,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 363, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:53:10,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:10,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 363, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:53:15,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:53:15,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:53:15,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.64 seconds 2025-02-15 15:53:15,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:15,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15498.15 MB 2025-02-15 15:53:15,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16782.79 MB 2025-02-15 15:53:15,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1284.64 MB 2025-02-15 15:53:15,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53856.96 MB 2025-02-15 15:53:15,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19073.60 MB 2025-02-15 15:53:15,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34783.36 MB 2025-02-15 15:53:15,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25649.00 MB 2025-02-15 15:53:15,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:53:15,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:53:15,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 15:53:15,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:15,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16782.79 MB 2025-02-15 15:53:15,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.49 MB 2025-02-15 15:53:15,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 623.71 MB 2025-02-15 15:53:15,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19073.60 MB 2025-02-15 15:53:15,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23534.24 MB 2025-02-15 15:53:15,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4460.64 MB 2025-02-15 15:53:15,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21884.24 MB 2025-02-15 15:53:17,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:53:17,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:53:17,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.75 seconds 2025-02-15 15:53:17,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.49 MB 2025-02-15 15:53:17,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17888.23 MB 2025-02-15 15:53:17,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.74 MB 2025-02-15 15:53:17,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23534.24 MB 2025-02-15 15:53:17,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20038.29 MB 2025-02-15 15:53:17,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3495.95 MB 2025-02-15 15:53:17,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21831.98 MB 2025-02-15 15:53:17,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:53:17,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:53:17,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:53:17,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17888.23 MB 2025-02-15 15:53:17,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19603.18 MB 2025-02-15 15:53:17,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.95 MB 2025-02-15 15:53:17,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20038.29 MB 2025-02-15 15:53:17,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-15 15:53:17,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3003.12 MB 2025-02-15 15:53:17,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20889.49 MB 2025-02-15 15:53:17,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:53:17,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:53:17,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 15:53:17,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19603.18 MB 2025-02-15 15:53:17,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.82 MB 2025-02-15 15:53:17,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2034.65 MB 2025-02-15 15:53:17,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-15 15:53:17,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 15:53:17,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5576.33 MB 2025-02-15 15:53:17,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26669.25 MB 2025-02-15 15:53:17,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:53:17,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:53:17,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:53:17,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17888.23 MB 2025-02-15 15:53:17,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.82 MB 2025-02-15 15:53:17,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3749.59 MB 2025-02-15 15:53:17,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20038.29 MB 2025-02-15 15:53:17,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28617.74 MB 2025-02-15 15:53:17,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8579.45 MB 2025-02-15 15:53:17,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26669.25 MB 2025-02-15 15:53:17,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:53:17,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:53:17,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-15 15:53:17,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23029.51 MB 2025-02-15 15:53:17,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23725.57 MB 2025-02-15 15:53:17,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 696.05 MB 2025-02-15 15:53:17,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28617.74 MB 2025-02-15 15:53:17,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-15 15:53:17,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 377.49 MB 2025-02-15 15:53:17,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24367.89 MB 2025-02-15 15:53:17,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:53:17,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:53:17,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:53:17,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24100.27 MB 2025-02-15 15:53:17,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24311.90 MB 2025-02-15 15:53:17,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.64 MB 2025-02-15 15:53:17,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28995.22 MB 2025-02-15 15:53:17,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-15 15:53:17,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:53:17,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24472.26 MB 2025-02-15 15:53:17,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:53:17,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:53:17,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.80 seconds 2025-02-15 15:53:17,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:17,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14233.43 MB 2025-02-15 15:53:17,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24512.97 MB 2025-02-15 15:53:17,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10279.55 MB 2025-02-15 15:53:17,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53856.96 MB 2025-02-15 15:53:17,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-15 15:53:17,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24861.74 MB 2025-02-15 15:53:17,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24512.97 MB 2025-02-15 15:53:18,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:53:18,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:53:18,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:53:18,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:18,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24512.97 MB 2025-02-15 15:53:18,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19062.81 MB 2025-02-15 15:53:18,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5450.16 MB 2025-02-15 15:53:18,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28995.22 MB 2025-02-15 15:53:18,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-15 15:53:18,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:53:18,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27526.97 MB 2025-02-15 15:53:18,087 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:53:18,087 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:53:18,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:53:18,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:53:18,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:53:18,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:18,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19062.81 MB 2025-02-15 15:53:18,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27501.83 MB 2025-02-15 15:53:18,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:53:18,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28995.22 MB 2025-02-15 15:53:18,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39485.18 MB 2025-02-15 15:53:18,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:53:18,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27501.83 MB 2025-02-15 15:53:18,256 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:53:18,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:18,258 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:53:18,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:18,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:53:18,263 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:53:18,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:18,265 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:53:18,265 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 15:53:43,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:43,856 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:53:43,863 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:53:43,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:43,869 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 696, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:53:43,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:43,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 696, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:53:54,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:53:54,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:53:54,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.91 seconds 2025-02-15 15:53:54,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:54,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17818.55 MB 2025-02-15 15:53:54,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20281.65 MB 2025-02-15 15:53:54,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2463.11 MB 2025-02-15 15:53:54,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52070.19 MB 2025-02-15 15:53:54,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24054.33 MB 2025-02-15 15:53:54,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28015.85 MB 2025-02-15 15:53:54,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29101.86 MB 2025-02-15 15:53:54,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:53:54,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:53:54,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 15:53:54,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:54,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20281.65 MB 2025-02-15 15:53:54,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19396.14 MB 2025-02-15 15:53:54,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -885.51 MB 2025-02-15 15:53:54,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24054.33 MB 2025-02-15 15:53:54,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30064.77 MB 2025-02-15 15:53:54,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6010.44 MB 2025-02-15 15:53:54,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29101.30 MB 2025-02-15 15:53:56,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:53:56,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:53:56,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 15:53:56,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:56,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19396.14 MB 2025-02-15 15:53:56,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19926.98 MB 2025-02-15 15:53:56,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:53:56,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30064.77 MB 2025-02-15 15:53:56,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23370.66 MB 2025-02-15 15:53:56,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6694.11 MB 2025-02-15 15:53:56,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23906.57 MB 2025-02-15 15:53:56,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:53:56,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:53:56,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:53:56,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:56,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19926.98 MB 2025-02-15 15:53:56,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21816.51 MB 2025-02-15 15:53:56,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:53:56,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23370.66 MB 2025-02-15 15:53:56,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25258.10 MB 2025-02-15 15:53:56,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:53:56,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23233.94 MB 2025-02-15 15:53:56,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:53:56,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:53:56,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:53:56,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:56,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21816.51 MB 2025-02-15 15:53:56,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24058.37 MB 2025-02-15 15:53:56,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:53:56,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25258.10 MB 2025-02-15 15:53:56,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-15 15:53:56,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-15 15:53:56,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29602.65 MB 2025-02-15 15:53:56,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:53:56,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:53:56,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:53:56,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:56,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19926.98 MB 2025-02-15 15:53:56,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24058.37 MB 2025-02-15 15:53:56,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:53:56,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23370.66 MB 2025-02-15 15:53:56,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-15 15:53:56,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-15 15:53:56,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29602.65 MB 2025-02-15 15:53:57,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:53:57,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:53:57,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:53:57,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:57,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25591.91 MB 2025-02-15 15:53:57,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26358.91 MB 2025-02-15 15:53:57,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:53:57,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31394.37 MB 2025-02-15 15:53:57,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31811.70 MB 2025-02-15 15:53:57,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:53:57,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27066.70 MB 2025-02-15 15:53:57,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:53:57,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:53:57,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:53:57,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:57,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26771.80 MB 2025-02-15 15:53:57,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26999.15 MB 2025-02-15 15:53:57,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.35 MB 2025-02-15 15:53:57,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31811.70 MB 2025-02-15 15:53:57,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31811.70 MB 2025-02-15 15:53:57,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:53:57,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27177.84 MB 2025-02-15 15:53:57,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:53:57,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:53:57,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.31 seconds 2025-02-15 15:53:57,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:57,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.63 MB 2025-02-15 15:53:57,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27200.01 MB 2025-02-15 15:53:57,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11806.38 MB 2025-02-15 15:53:57,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52070.19 MB 2025-02-15 15:53:57,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31811.70 MB 2025-02-15 15:53:57,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20258.49 MB 2025-02-15 15:53:57,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27200.01 MB 2025-02-15 15:53:57,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:53:57,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:53:57,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:53:57,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:57,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27200.01 MB 2025-02-15 15:53:57,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20384.25 MB 2025-02-15 15:53:57,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6815.75 MB 2025-02-15 15:53:57,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31811.70 MB 2025-02-15 15:53:57,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31811.70 MB 2025-02-15 15:53:57,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:53:57,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29700.00 MB 2025-02-15 15:53:57,475 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-15 15:53:57,475 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:53:57,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:53:57,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:53:57,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:53:57,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:53:57,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20384.25 MB 2025-02-15 15:53:57,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28785.11 MB 2025-02-15 15:53:57,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-15 15:53:57,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31811.70 MB 2025-02-15 15:53:57,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40162.56 MB 2025-02-15 15:53:57,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-15 15:53:57,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28785.11 MB 2025-02-15 15:53:57,640 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-15 15:53:57,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:57,642 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:53:57,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:57,643 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:53:57,648 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:53:57,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:53:57,649 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:53:57,649 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:54:46,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:46,187 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:54:46,192 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:54:46,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:46,196 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 474, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:54:46,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:46,197 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 474, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:54:53,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:54:53,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:54:53,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.32 seconds 2025-02-15 15:54:53,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:53,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16271.61 MB 2025-02-15 15:54:53,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17949.07 MB 2025-02-15 15:54:53,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1677.46 MB 2025-02-15 15:54:53,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 15:54:53,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20252.20 MB 2025-02-15 15:54:53,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28261.22 MB 2025-02-15 15:54:53,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26875.45 MB 2025-02-15 15:54:53,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:54:53,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:54:53,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 15:54:53,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:53,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17949.07 MB 2025-02-15 15:54:53,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18243.08 MB 2025-02-15 15:54:53,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.00 MB 2025-02-15 15:54:53,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20252.20 MB 2025-02-15 15:54:53,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25660.75 MB 2025-02-15 15:54:53,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5408.56 MB 2025-02-15 15:54:53,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25455.05 MB 2025-02-15 15:54:55,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:54:55,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:54:55,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:54:55,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18243.08 MB 2025-02-15 15:54:55,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18773.92 MB 2025-02-15 15:54:55,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:54:55,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25660.75 MB 2025-02-15 15:54:55,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21432.89 MB 2025-02-15 15:54:55,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4227.86 MB 2025-02-15 15:54:55,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22753.50 MB 2025-02-15 15:54:55,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:54:55,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:54:55,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:54:55,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-15 15:54:55,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20663.45 MB 2025-02-15 15:54:55,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:54:55,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 15:54:55,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24264.05 MB 2025-02-15 15:54:55,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 15:54:55,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22080.88 MB 2025-02-15 15:54:55,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:54:55,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:54:55,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:54:55,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20663.45 MB 2025-02-15 15:54:55,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-15 15:54:55,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:54:55,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24264.05 MB 2025-02-15 15:54:55,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 15:54:55,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:54:55,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-15 15:54:55,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:54:55,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:54:55,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:54:55,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-15 15:54:55,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-15 15:54:55,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:54:55,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21432.89 MB 2025-02-15 15:54:55,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30398.22 MB 2025-02-15 15:54:55,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 15:54:55,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-15 15:54:55,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:54:55,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:54:55,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:54:55,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24438.85 MB 2025-02-15 15:54:55,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25205.85 MB 2025-02-15 15:54:55,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:54:55,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30398.22 MB 2025-02-15 15:54:55,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 15:54:55,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 15:54:55,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25913.64 MB 2025-02-15 15:54:55,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:54:55,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:54:55,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:54:55,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25618.74 MB 2025-02-15 15:54:55,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25847.68 MB 2025-02-15 15:54:55,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-15 15:54:55,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 15:54:55,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 15:54:55,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:54:55,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26068.83 MB 2025-02-15 15:54:55,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:54:55,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:54:55,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.72 seconds 2025-02-15 15:54:55,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:55,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14620.16 MB 2025-02-15 15:54:55,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.75 MB 2025-02-15 15:54:55,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11428.59 MB 2025-02-15 15:54:55,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48513.42 MB 2025-02-15 15:54:55,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 15:54:55,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17697.87 MB 2025-02-15 15:54:55,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26068.83 MB 2025-02-15 15:54:56,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:54:56,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:54:56,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:54:56,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:56,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.75 MB 2025-02-15 15:54:56,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19624.55 MB 2025-02-15 15:54:56,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6424.20 MB 2025-02-15 15:54:56,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 15:54:56,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-15 15:54:56,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:54:56,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28560.42 MB 2025-02-15 15:54:56,206 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:54:56,206 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:54:56,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:54:56,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:54:56,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:54:56,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:54:56,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19624.55 MB 2025-02-15 15:54:56,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28063.57 MB 2025-02-15 15:54:56,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:54:56,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-15 15:54:56,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41305.51 MB 2025-02-15 15:54:56,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:54:56,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28063.57 MB 2025-02-15 15:54:56,374 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:54:56,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:56,376 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:54:56,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:56,377 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:54:56,381 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:54:56,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:54:56,382 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:54:56,383 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:55:05,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:05,372 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:55:05,377 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:55:05,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:05,381 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:55:05,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:05,381 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:55:23,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:55:23,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:55:23,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.83 seconds 2025-02-15 15:55:23,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:23,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20961.19 MB 2025-02-15 15:55:23,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.27 MB 2025-02-15 15:55:23,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4060.09 MB 2025-02-15 15:55:23,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 15:55:23,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26835.16 MB 2025-02-15 15:55:23,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27055.36 MB 2025-02-15 15:55:23,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33830.75 MB 2025-02-15 15:55:23,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:55:23,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:55:23,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 15:55:23,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:23,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25021.27 MB 2025-02-15 15:55:23,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21741.79 MB 2025-02-15 15:55:23,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3279.48 MB 2025-02-15 15:55:23,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26835.16 MB 2025-02-15 15:55:23,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36651.93 MB 2025-02-15 15:55:23,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9816.77 MB 2025-02-15 15:55:23,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36937.91 MB 2025-02-15 15:55:25,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:55:25,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:55:25,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:55:25,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21741.79 MB 2025-02-15 15:55:25,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22272.64 MB 2025-02-15 15:55:25,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:55:25,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36651.93 MB 2025-02-15 15:55:25,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24899.49 MB 2025-02-15 15:55:25,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11752.44 MB 2025-02-15 15:55:25,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26252.22 MB 2025-02-15 15:55:25,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:55:25,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:55:25,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:55:25,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22272.64 MB 2025-02-15 15:55:25,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24162.17 MB 2025-02-15 15:55:25,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 15:55:25,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24899.49 MB 2025-02-15 15:55:25,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27730.64 MB 2025-02-15 15:55:25,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 15:55:25,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25579.60 MB 2025-02-15 15:55:25,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:55:25,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:55:25,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:55:25,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24162.17 MB 2025-02-15 15:55:25,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26404.03 MB 2025-02-15 15:55:25,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:55:25,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27730.64 MB 2025-02-15 15:55:25,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33864.81 MB 2025-02-15 15:55:25,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:55:25,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31948.31 MB 2025-02-15 15:55:25,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:55:25,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:55:25,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 15:55:25,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22272.64 MB 2025-02-15 15:55:25,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26404.03 MB 2025-02-15 15:55:25,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 15:55:25,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24899.49 MB 2025-02-15 15:55:25,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33864.81 MB 2025-02-15 15:55:25,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 15:55:25,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31948.31 MB 2025-02-15 15:55:25,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:55:25,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:55:25,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:55:25,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27937.57 MB 2025-02-15 15:55:25,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28704.57 MB 2025-02-15 15:55:25,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:55:25,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33864.81 MB 2025-02-15 15:55:25,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 15:55:25,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-15 15:55:25,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29412.36 MB 2025-02-15 15:55:25,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:55:25,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:55:25,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:55:25,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29117.46 MB 2025-02-15 15:55:25,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29346.68 MB 2025-02-15 15:55:25,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.22 MB 2025-02-15 15:55:25,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 15:55:25,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 15:55:25,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:25,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29554.77 MB 2025-02-15 15:55:25,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:55:25,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:55:25,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.31 seconds 2025-02-15 15:55:25,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16964.95 MB 2025-02-15 15:55:25,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29547.54 MB 2025-02-15 15:55:25,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12582.59 MB 2025-02-15 15:55:25,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53890.51 MB 2025-02-15 15:55:25,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 15:55:25,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-15 15:55:25,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29554.77 MB 2025-02-15 15:55:25,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:55:25,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:55:25,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:55:25,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29547.54 MB 2025-02-15 15:55:25,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21959.69 MB 2025-02-15 15:55:25,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7587.85 MB 2025-02-15 15:55:25,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 15:55:25,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34277.95 MB 2025-02-15 15:55:25,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:25,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32051.20 MB 2025-02-15 15:55:25,984 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 15:55:25,984 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:55:25,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:55:25,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:55:25,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:55:25,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:25,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21959.69 MB 2025-02-15 15:55:25,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30368.99 MB 2025-02-15 15:55:25,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-15 15:55:25,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34277.95 MB 2025-02-15 15:55:25,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42637.20 MB 2025-02-15 15:55:25,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 15:55:25,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30368.99 MB 2025-02-15 15:55:26,154 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 15:55:26,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:26,155 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:55:26,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:26,156 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:55:26,161 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:55:26,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:26,162 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:55:26,162 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:55:34,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:34,015 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:55:34,021 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:55:34,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:34,025 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:55:34,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:34,027 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:55:36,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:55:36,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:55:36,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-15 15:55:36,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:36,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-15 15:55:36,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-15 15:55:36,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-15 15:55:36,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50996.45 MB 2025-02-15 15:55:36,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 15:55:36,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32392.61 MB 2025-02-15 15:55:36,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23499.24 MB 2025-02-15 15:55:36,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:55:36,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:55:36,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:55:36,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:36,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-15 15:55:36,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14699.99 MB 2025-02-15 15:55:36,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.21 MB 2025-02-15 15:55:36,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 15:55:36,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 15:55:36,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:36,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16448.02 MB 2025-02-15 15:55:37,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:55:37,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:55:37,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-15 15:55:37,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14699.99 MB 2025-02-15 15:55:37,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14877.83 MB 2025-02-15 15:55:37,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-15 15:55:37,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 15:55:37,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 15:55:37,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:37,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18871.12 MB 2025-02-15 15:55:37,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:55:37,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:55:37,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 15:55:37,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-15 15:55:37,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15510.60 MB 2025-02-15 15:55:37,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-15 15:55:37,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 15:55:37,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-15 15:55:37,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:37,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15985.44 MB 2025-02-15 15:55:37,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:55:37,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:55:37,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 15:55:37,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15510.60 MB 2025-02-15 15:55:37,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-15 15:55:37,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-15 15:55:37,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 15:55:37,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-15 15:55:37,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 633.34 MB 2025-02-15 15:55:37,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18120.00 MB 2025-02-15 15:55:37,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:55:37,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:55:37,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 15:55:37,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-15 15:55:37,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-15 15:55:37,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-15 15:55:37,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-15 15:55:37,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-15 15:55:37,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 633.34 MB 2025-02-15 15:55:37,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18120.00 MB 2025-02-15 15:55:37,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:55:37,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:55:37,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 15:55:37,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16775.40 MB 2025-02-15 15:55:37,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17032.35 MB 2025-02-15 15:55:37,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-15 15:55:37,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-15 15:55:37,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-15 15:55:37,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-15 15:55:37,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17279.96 MB 2025-02-15 15:55:37,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:55:37,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:55:37,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:55:37,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17170.67 MB 2025-02-15 15:55:37,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17389.83 MB 2025-02-15 15:55:37,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.16 MB 2025-02-15 15:55:37,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-15 15:55:37,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-15 15:55:37,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:37,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17389.83 MB 2025-02-15 15:55:37,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:55:37,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:55:37,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-15 15:55:37,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-15 15:55:37,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14233.11 MB 2025-02-15 15:55:37,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.83 MB 2025-02-15 15:55:37,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50996.45 MB 2025-02-15 15:55:37,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-15 15:55:37,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31622.96 MB 2025-02-15 15:55:37,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17590.51 MB 2025-02-15 15:55:37,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:55:37,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:55:37,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:55:37,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14233.11 MB 2025-02-15 15:55:37,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17241.25 MB 2025-02-15 15:55:37,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.14 MB 2025-02-15 15:55:37,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-15 15:55:37,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-15 15:55:37,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:55:37,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17542.03 MB 2025-02-15 15:55:37,538 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 15:55:37,539 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:55:37,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:55:37,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:55:37,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:55:37,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:55:37,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17241.25 MB 2025-02-15 15:55:37,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25663.58 MB 2025-02-15 15:55:37,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 15:55:37,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-15 15:55:37,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29842.47 MB 2025-02-15 15:55:37,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 15:55:37,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25663.58 MB 2025-02-15 15:55:37,707 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 15:55:37,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:37,708 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:55:37,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:37,709 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:55:37,714 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:55:37,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:55:37,715 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:55:37,715 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 15:56:16,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:16,130 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:56:16,135 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:56:16,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:16,140 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:56:16,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:16,141 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:56:18,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:56:18,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:56:18,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-15 15:56:18,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:18,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19433.12 MB 2025-02-15 15:56:18,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19974.58 MB 2025-02-15 15:56:18,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-15 15:56:18,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42402.32 MB 2025-02-15 15:56:18,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22716.35 MB 2025-02-15 15:56:18,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19685.97 MB 2025-02-15 15:56:18,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28905.30 MB 2025-02-15 15:56:18,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:56:18,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:56:18,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:56:18,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:18,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19974.58 MB 2025-02-15 15:56:18,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20194.78 MB 2025-02-15 15:56:18,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-15 15:56:18,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22716.35 MB 2025-02-15 15:56:18,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23754.44 MB 2025-02-15 15:56:18,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1038.09 MB 2025-02-15 15:56:18,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22039.41 MB 2025-02-15 15:56:19,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:56:19,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:56:19,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.74 seconds 2025-02-15 15:56:19,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20194.78 MB 2025-02-15 15:56:19,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20389.86 MB 2025-02-15 15:56:19,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-15 15:56:19,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23754.44 MB 2025-02-15 15:56:19,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23754.44 MB 2025-02-15 15:56:19,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:56:19,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24365.47 MB 2025-02-15 15:56:19,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:56:19,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:56:19,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:56:19,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20389.80 MB 2025-02-15 15:56:19,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21084.03 MB 2025-02-15 15:56:19,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-15 15:56:19,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23754.44 MB 2025-02-15 15:56:19,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23754.44 MB 2025-02-15 15:56:19,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:56:19,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21604.94 MB 2025-02-15 15:56:19,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:56:19,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:56:19,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 15:56:19,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21084.03 MB 2025-02-15 15:56:19,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21907.96 MB 2025-02-15 15:56:19,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-15 15:56:19,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23754.44 MB 2025-02-15 15:56:19,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25146.95 MB 2025-02-15 15:56:19,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 15:56:19,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23945.44 MB 2025-02-15 15:56:19,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:56:19,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:56:19,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 15:56:19,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20389.80 MB 2025-02-15 15:56:19,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21907.96 MB 2025-02-15 15:56:19,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-15 15:56:19,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23754.44 MB 2025-02-15 15:56:19,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25146.95 MB 2025-02-15 15:56:19,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-15 15:56:19,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23945.44 MB 2025-02-15 15:56:19,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:56:19,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:56:19,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 15:56:19,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22471.53 MB 2025-02-15 15:56:19,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22753.41 MB 2025-02-15 15:56:19,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-15 15:56:19,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25146.95 MB 2025-02-15 15:56:19,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25295.85 MB 2025-02-15 15:56:19,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-15 15:56:19,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23025.79 MB 2025-02-15 15:56:19,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:56:19,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:56:19,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:56:19,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22905.15 MB 2025-02-15 15:56:19,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23112.79 MB 2025-02-15 15:56:19,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.64 MB 2025-02-15 15:56:19,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25295.85 MB 2025-02-15 15:56:19,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-15 15:56:19,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 15:56:19,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23114.95 MB 2025-02-15 15:56:19,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:56:19,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:56:19,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-15 15:56:19,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18900.06 MB 2025-02-15 15:56:19,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23313.42 MB 2025-02-15 15:56:19,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4413.36 MB 2025-02-15 15:56:19,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42402.32 MB 2025-02-15 15:56:19,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-15 15:56:19,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17104.37 MB 2025-02-15 15:56:19,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23313.42 MB 2025-02-15 15:56:19,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:56:19,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:56:19,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 15:56:19,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23313.42 MB 2025-02-15 15:56:19,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22703.22 MB 2025-02-15 15:56:19,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -610.20 MB 2025-02-15 15:56:19,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-15 15:56:19,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-15 15:56:19,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:56:19,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24315.88 MB 2025-02-15 15:56:19,843 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-15 15:56:19,844 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:56:19,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:56:19,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:56:19,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:56:19,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:19,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22703.22 MB 2025-02-15 15:56:19,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31124.00 MB 2025-02-15 15:56:19,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-15 15:56:19,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-15 15:56:19,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35762.73 MB 2025-02-15 15:56:19,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-15 15:56:19,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31124.00 MB 2025-02-15 15:56:20,102 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-15 15:56:20,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:20,105 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:56:20,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:20,107 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:56:20,114 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:56:20,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:20,116 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:56:20,116 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 15:56:47,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:47,774 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:56:47,779 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:56:47,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:47,783 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 770, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:56:47,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:56:47,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 770, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:56:59,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:56:59,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:56:59,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.97 seconds 2025-02-15 15:56:59,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:59,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23732.48 MB 2025-02-15 15:56:59,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26457.46 MB 2025-02-15 15:56:59,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2724.99 MB 2025-02-15 15:56:59,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-15 15:56:59,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31404.85 MB 2025-02-15 15:56:59,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12729.71 MB 2025-02-15 15:56:59,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35469.58 MB 2025-02-15 15:56:59,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:56:59,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:56:59,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-15 15:56:59,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:56:59,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26457.46 MB 2025-02-15 15:56:59,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25179.13 MB 2025-02-15 15:56:59,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1278.34 MB 2025-02-15 15:56:59,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31404.85 MB 2025-02-15 15:56:59,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37612.42 MB 2025-02-15 15:56:59,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6207.57 MB 2025-02-15 15:56:59,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35473.06 MB 2025-02-15 15:57:01,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:57:01,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:57:01,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 15:57:01,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:01,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25179.13 MB 2025-02-15 15:57:01,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25709.97 MB 2025-02-15 15:57:01,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:57:01,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37612.42 MB 2025-02-15 15:57:01,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30094.13 MB 2025-02-15 15:57:01,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7518.29 MB 2025-02-15 15:57:01,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29688.52 MB 2025-02-15 15:57:01,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:57:01,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:57:01,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:57:01,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:01,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25709.97 MB 2025-02-15 15:57:01,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27599.14 MB 2025-02-15 15:57:01,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 15:57:01,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30094.13 MB 2025-02-15 15:57:01,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31981.57 MB 2025-02-15 15:57:01,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:57:01,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29016.57 MB 2025-02-15 15:57:01,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:57:01,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:57:01,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:57:01,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:01,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27599.14 MB 2025-02-15 15:57:01,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29841.00 MB 2025-02-15 15:57:01,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:57:01,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31981.57 MB 2025-02-15 15:57:01,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38115.74 MB 2025-02-15 15:57:01,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 15:57:01,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35385.28 MB 2025-02-15 15:57:01,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:57:01,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:57:01,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:57:01,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:01,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25709.97 MB 2025-02-15 15:57:01,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29841.00 MB 2025-02-15 15:57:01,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 15:57:01,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30094.13 MB 2025-02-15 15:57:01,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38115.74 MB 2025-02-15 15:57:01,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 15:57:01,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35385.28 MB 2025-02-15 15:57:02,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:57:02,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:57:02,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:57:02,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:02,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31374.54 MB 2025-02-15 15:57:02,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32141.54 MB 2025-02-15 15:57:02,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:57:02,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38115.74 MB 2025-02-15 15:57:02,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38530.97 MB 2025-02-15 15:57:02,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:57:02,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32849.33 MB 2025-02-15 15:57:02,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:57:02,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:57:02,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:57:02,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:02,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32554.43 MB 2025-02-15 15:57:02,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32781.54 MB 2025-02-15 15:57:02,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.11 MB 2025-02-15 15:57:02,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38530.97 MB 2025-02-15 15:57:02,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38530.97 MB 2025-02-15 15:57:02,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:57:02,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32969.56 MB 2025-02-15 15:57:02,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:57:02,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:57:02,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.36 seconds 2025-02-15 15:57:02,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:02,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21049.74 MB 2025-02-15 15:57:02,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32982.22 MB 2025-02-15 15:57:02,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11932.48 MB 2025-02-15 15:57:02,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-15 15:57:02,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38530.97 MB 2025-02-15 15:57:02,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5603.59 MB 2025-02-15 15:57:02,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32982.22 MB 2025-02-15 15:57:02,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:57:02,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:57:02,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:57:02,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:02,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32982.22 MB 2025-02-15 15:57:02,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.03 MB 2025-02-15 15:57:02,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6934.19 MB 2025-02-15 15:57:02,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38530.97 MB 2025-02-15 15:57:02,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38530.97 MB 2025-02-15 15:57:02,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:57:02,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35488.97 MB 2025-02-15 15:57:02,429 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 15:57:02,429 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:57:02,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:57:02,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:57:02,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:57:02,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:57:02,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.03 MB 2025-02-15 15:57:02,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34470.36 MB 2025-02-15 15:57:02,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 15:57:02,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38530.97 MB 2025-02-15 15:57:02,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48999.96 MB 2025-02-15 15:57:02,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-15 15:57:02,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34470.36 MB 2025-02-15 15:57:02,593 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 15:57:02,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:57:02,594 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:57:02,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:57:02,595 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:57:02,600 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:57:02,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:57:02,601 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:57:02,601 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:58:07,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:07,864 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:58:07,869 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:58:07,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:07,873 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 508, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:58:07,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:07,874 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 508, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:58:15,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:58:15,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:58:15,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.80 seconds 2025-02-15 15:58:15,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:15,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21906.82 MB 2025-02-15 15:58:15,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23704.60 MB 2025-02-15 15:58:15,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1797.78 MB 2025-02-15 15:58:15,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61559.80 MB 2025-02-15 15:58:15,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27973.91 MB 2025-02-15 15:58:15,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33585.89 MB 2025-02-15 15:58:15,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32511.46 MB 2025-02-15 15:58:15,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:58:15,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:58:15,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-15 15:58:15,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:15,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23704.60 MB 2025-02-15 15:58:15,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23818.12 MB 2025-02-15 15:58:15,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 113.52 MB 2025-02-15 15:58:15,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27973.91 MB 2025-02-15 15:58:15,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33470.55 MB 2025-02-15 15:58:15,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5496.64 MB 2025-02-15 15:58:15,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31301.49 MB 2025-02-15 15:58:17,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:58:17,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:58:17,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 15:58:17,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:17,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23818.12 MB 2025-02-15 15:58:17,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.96 MB 2025-02-15 15:58:17,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:58:17,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33470.55 MB 2025-02-15 15:58:17,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27355.25 MB 2025-02-15 15:58:17,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6115.30 MB 2025-02-15 15:58:17,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28328.55 MB 2025-02-15 15:58:17,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:58:17,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:58:17,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:58:17,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:17,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.96 MB 2025-02-15 15:58:17,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26238.14 MB 2025-02-15 15:58:17,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 15:58:17,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27355.25 MB 2025-02-15 15:58:17,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29242.69 MB 2025-02-15 15:58:17,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 15:58:17,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27655.56 MB 2025-02-15 15:58:17,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:58:17,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:58:17,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:58:17,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:17,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26238.14 MB 2025-02-15 15:58:17,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28479.99 MB 2025-02-15 15:58:17,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:58:17,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29242.69 MB 2025-02-15 15:58:17,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36320.58 MB 2025-02-15 15:58:17,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 15:58:17,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34024.27 MB 2025-02-15 15:58:17,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:58:17,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:58:17,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:58:17,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:17,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.96 MB 2025-02-15 15:58:17,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28479.99 MB 2025-02-15 15:58:17,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 15:58:17,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27355.25 MB 2025-02-15 15:58:17,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36320.58 MB 2025-02-15 15:58:17,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-15 15:58:17,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34024.27 MB 2025-02-15 15:58:18,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:58:18,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:58:18,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 15:58:18,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:18,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30014.14 MB 2025-02-15 15:58:18,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30781.14 MB 2025-02-15 15:58:18,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:58:18,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36320.58 MB 2025-02-15 15:58:18,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36735.81 MB 2025-02-15 15:58:18,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:58:18,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31488.93 MB 2025-02-15 15:58:18,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:58:18,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:58:18,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:58:18,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:18,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31194.03 MB 2025-02-15 15:58:18,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31420.71 MB 2025-02-15 15:58:18,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.69 MB 2025-02-15 15:58:18,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36735.81 MB 2025-02-15 15:58:18,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36735.81 MB 2025-02-15 15:58:18,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:58:18,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31585.87 MB 2025-02-15 15:58:18,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:58:18,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:58:18,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.18 seconds 2025-02-15 15:58:18,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:18,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20136.91 MB 2025-02-15 15:58:18,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31621.79 MB 2025-02-15 15:58:18,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11484.88 MB 2025-02-15 15:58:18,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61559.80 MB 2025-02-15 15:58:18,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36735.81 MB 2025-02-15 15:58:18,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24823.99 MB 2025-02-15 15:58:18,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31621.79 MB 2025-02-15 15:58:18,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:58:18,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:58:18,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:58:18,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:18,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31621.79 MB 2025-02-15 15:58:18,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25141.90 MB 2025-02-15 15:58:18,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6479.89 MB 2025-02-15 15:58:18,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36735.81 MB 2025-02-15 15:58:18,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36735.81 MB 2025-02-15 15:58:18,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:58:18,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34133.45 MB 2025-02-15 15:58:18,344 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:58:18,344 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:58:18,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:58:18,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:58:18,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:58:18,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:18,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25141.90 MB 2025-02-15 15:58:18,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33580.92 MB 2025-02-15 15:58:18,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:58:18,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36735.81 MB 2025-02-15 15:58:18,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47225.77 MB 2025-02-15 15:58:18,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:58:18,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33580.92 MB 2025-02-15 15:58:18,512 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:58:18,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:18,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:58:18,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:18,514 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:58:18,519 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:58:18,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:18,520 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:58:18,520 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:58:31,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:31,731 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:58:31,739 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:58:31,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:31,746 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1498, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:58:31,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:31,748 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1498, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:58:55,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:58:55,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:58:55,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.39 seconds 2025-02-15 15:58:55,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:55,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28805.30 MB 2025-02-15 15:58:55,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34106.90 MB 2025-02-15 15:58:55,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5301.60 MB 2025-02-15 15:58:55,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59810.78 MB 2025-02-15 15:58:55,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44480.59 MB 2025-02-15 15:58:55,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15330.18 MB 2025-02-15 15:58:55,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43033.82 MB 2025-02-15 15:58:55,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:58:55,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:58:55,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 15:58:55,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:55,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34106.90 MB 2025-02-15 15:58:55,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28963.77 MB 2025-02-15 15:58:55,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5143.12 MB 2025-02-15 15:58:55,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44480.59 MB 2025-02-15 15:58:55,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54007.96 MB 2025-02-15 15:58:55,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9527.36 MB 2025-02-15 15:58:55,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48431.54 MB 2025-02-15 15:58:57,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:58:57,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:58:57,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 15:58:57,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28963.77 MB 2025-02-15 15:58:57,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29494.61 MB 2025-02-15 15:58:57,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 15:58:57,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54007.96 MB 2025-02-15 15:58:57,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 15:58:57,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14828.96 MB 2025-02-15 15:58:57,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33473.16 MB 2025-02-15 15:58:57,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:58:57,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:58:57,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:58:57,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29494.61 MB 2025-02-15 15:58:57,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31383.79 MB 2025-02-15 15:58:57,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 15:58:57,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 15:58:57,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 15:58:57,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:58:57,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32801.22 MB 2025-02-15 15:58:57,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:58:57,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:58:57,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 15:58:57,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31383.79 MB 2025-02-15 15:58:57,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33625.64 MB 2025-02-15 15:58:57,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 15:58:57,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 15:58:57,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 15:58:57,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 15:58:57,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39169.93 MB 2025-02-15 15:58:57,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:58:57,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:58:57,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 15:58:57,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29494.61 MB 2025-02-15 15:58:57,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33625.64 MB 2025-02-15 15:58:57,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 15:58:57,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 15:58:57,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-15 15:58:57,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-15 15:58:57,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39169.93 MB 2025-02-15 15:58:57,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:58:57,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:58:57,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 15:58:57,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35159.79 MB 2025-02-15 15:58:57,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35926.79 MB 2025-02-15 15:58:57,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 15:58:57,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42953.87 MB 2025-02-15 15:58:57,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43369.10 MB 2025-02-15 15:58:57,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 15:58:57,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36634.58 MB 2025-02-15 15:58:57,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:58:57,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:58:57,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:58:57,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36339.68 MB 2025-02-15 15:58:57,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36568.46 MB 2025-02-15 15:58:57,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.78 MB 2025-02-15 15:58:57,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43369.10 MB 2025-02-15 15:58:57,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43369.10 MB 2025-02-15 15:58:57,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:58:57,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36766.35 MB 2025-02-15 15:58:57,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:58:57,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:58:57,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.86 seconds 2025-02-15 15:58:57,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23586.15 MB 2025-02-15 15:58:57,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36769.53 MB 2025-02-15 15:58:57,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13183.38 MB 2025-02-15 15:58:57,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59810.78 MB 2025-02-15 15:58:57,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43369.10 MB 2025-02-15 15:58:57,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16441.67 MB 2025-02-15 15:58:57,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36769.53 MB 2025-02-15 15:58:57,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:58:57,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:58:57,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:58:57,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36769.53 MB 2025-02-15 15:58:57,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28591.14 MB 2025-02-15 15:58:57,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8178.39 MB 2025-02-15 15:58:57,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43369.10 MB 2025-02-15 15:58:57,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43369.10 MB 2025-02-15 15:58:57,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:58:57,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39281.20 MB 2025-02-15 15:58:57,896 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:58:57,896 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:58:57,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:58:57,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:58:57,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:58:57,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:58:57,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28591.14 MB 2025-02-15 15:58:57,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37030.16 MB 2025-02-15 15:58:57,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:58:57,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43369.10 MB 2025-02-15 15:58:57,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51759.81 MB 2025-02-15 15:58:57,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 15:58:57,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37030.16 MB 2025-02-15 15:58:58,062 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:58:58,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:58,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:58:58,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:58,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:58:58,069 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:58:58,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:58:58,070 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:58:58,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 15:59:47,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:47,990 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 15:59:47,996 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 15:59:48,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:48,001 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 15:59:48,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:48,002 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 15:59:51,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 15:59:51,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 15:59:51,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.27 seconds 2025-02-15 15:59:51,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:51,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19823.34 MB 2025-02-15 15:59:51,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20562.98 MB 2025-02-15 15:59:51,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.64 MB 2025-02-15 15:59:51,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64344.82 MB 2025-02-15 15:59:51,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23345.50 MB 2025-02-15 15:59:51,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40999.32 MB 2025-02-15 15:59:51,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29522.01 MB 2025-02-15 15:59:51,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 15:59:51,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 15:59:51,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:59:51,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:51,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20562.98 MB 2025-02-15 15:59:51,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20922.18 MB 2025-02-15 15:59:51,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 359.20 MB 2025-02-15 15:59:51,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23345.50 MB 2025-02-15 15:59:51,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25555.89 MB 2025-02-15 15:59:51,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2210.40 MB 2025-02-15 15:59:51,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23500.17 MB 2025-02-15 15:59:52,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 15:59:52,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 15:59:52,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-15 15:59:52,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20922.18 MB 2025-02-15 15:59:52,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21199.55 MB 2025-02-15 15:59:52,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-15 15:59:52,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25555.89 MB 2025-02-15 15:59:52,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25218.25 MB 2025-02-15 15:59:52,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -337.64 MB 2025-02-15 15:59:52,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25177.80 MB 2025-02-15 15:59:52,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 15:59:52,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 15:59:52,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:59:52,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21199.55 MB 2025-02-15 15:59:52,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22186.59 MB 2025-02-15 15:59:52,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-15 15:59:52,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25218.25 MB 2025-02-15 15:59:52,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25218.25 MB 2025-02-15 15:59:52,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:59:52,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22927.20 MB 2025-02-15 15:59:52,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 15:59:52,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 15:59:52,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 15:59:52,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22186.59 MB 2025-02-15 15:59:52,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23357.99 MB 2025-02-15 15:59:52,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.40 MB 2025-02-15 15:59:52,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25218.25 MB 2025-02-15 15:59:52,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28187.82 MB 2025-02-15 15:59:52,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-15 15:59:52,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26256.68 MB 2025-02-15 15:59:52,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 15:59:52,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 15:59:52,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 15:59:52,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21199.55 MB 2025-02-15 15:59:52,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23357.99 MB 2025-02-15 15:59:52,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.44 MB 2025-02-15 15:59:52,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25218.25 MB 2025-02-15 15:59:52,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28187.82 MB 2025-02-15 15:59:52,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-15 15:59:52,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26256.68 MB 2025-02-15 15:59:52,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 15:59:52,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 15:59:52,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 15:59:52,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.27 MB 2025-02-15 15:59:52,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24561.86 MB 2025-02-15 15:59:52,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.59 MB 2025-02-15 15:59:52,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28187.82 MB 2025-02-15 15:59:52,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-15 15:59:52,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-15 15:59:52,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24933.36 MB 2025-02-15 15:59:52,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 15:59:52,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 15:59:52,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 15:59:52,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24777.60 MB 2025-02-15 15:59:52,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25006.13 MB 2025-02-15 15:59:52,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-15 15:59:52,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28401.73 MB 2025-02-15 15:59:52,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-15 15:59:52,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:59:52,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25072.70 MB 2025-02-15 15:59:52,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 15:59:52,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 15:59:52,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.52 seconds 2025-02-15 15:59:52,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19095.17 MB 2025-02-15 15:59:52,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25207.21 MB 2025-02-15 15:59:52,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6112.04 MB 2025-02-15 15:59:52,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64344.82 MB 2025-02-15 15:59:52,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-15 15:59:52,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35943.09 MB 2025-02-15 15:59:52,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25207.21 MB 2025-02-15 15:59:52,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 15:59:52,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 15:59:52,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 15:59:52,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20185.45 MB 2025-02-15 15:59:52,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23200.27 MB 2025-02-15 15:59:52,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.82 MB 2025-02-15 15:59:52,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28401.73 MB 2025-02-15 15:59:52,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-15 15:59:52,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 15:59:52,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23501.64 MB 2025-02-15 15:59:52,815 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 15:59:52,815 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 15:59:52,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 15:59:52,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 15:59:52,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 15:59:52,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 15:59:52,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23200.27 MB 2025-02-15 15:59:52,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31639.29 MB 2025-02-15 15:59:52,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 15:59:52,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28401.73 MB 2025-02-15 15:59:52,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38891.68 MB 2025-02-15 15:59:52,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 15:59:52,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31639.29 MB 2025-02-15 15:59:52,983 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 15:59:52,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:52,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 15:59:52,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:52,986 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 15:59:52,991 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 15:59:52,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 15:59:52,992 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 15:59:52,992 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 16:00:02,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:02,000 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:00:02,005 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:00:02,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:02,009 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1250, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:00:02,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:02,010 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1250, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:00:21,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:00:21,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:00:21,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.39 seconds 2025-02-15 16:00:21,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:21,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27077.19 MB 2025-02-15 16:00:21,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31500.87 MB 2025-02-15 16:00:21,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4423.68 MB 2025-02-15 16:00:21,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51476.69 MB 2025-02-15 16:00:21,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43603.98 MB 2025-02-15 16:00:21,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7872.71 MB 2025-02-15 16:00:21,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40398.94 MB 2025-02-15 16:00:21,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:00:21,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:00:21,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 16:00:21,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:21,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31500.87 MB 2025-02-15 16:00:21,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27674.50 MB 2025-02-15 16:00:21,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3826.38 MB 2025-02-15 16:00:21,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43603.98 MB 2025-02-15 16:00:21,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52328.14 MB 2025-02-15 16:00:21,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8724.15 MB 2025-02-15 16:00:21,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44619.27 MB 2025-02-15 16:00:23,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:00:23,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:00:23,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 16:00:23,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27674.50 MB 2025-02-15 16:00:23,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28205.34 MB 2025-02-15 16:00:23,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 16:00:23,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52328.14 MB 2025-02-15 16:00:23,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 16:00:23,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13149.14 MB 2025-02-15 16:00:23,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32183.89 MB 2025-02-15 16:00:23,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:00:23,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:00:23,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:00:23,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28205.34 MB 2025-02-15 16:00:23,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30094.51 MB 2025-02-15 16:00:23,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 16:00:23,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:00:23,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 16:00:23,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:00:23,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31511.94 MB 2025-02-15 16:00:23,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:00:23,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:00:23,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 16:00:23,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30094.51 MB 2025-02-15 16:00:23,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32336.97 MB 2025-02-15 16:00:23,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.46 MB 2025-02-15 16:00:23,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:00:23,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-15 16:00:23,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 16:00:23,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37881.25 MB 2025-02-15 16:00:23,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:00:23,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:00:23,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 16:00:23,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28205.34 MB 2025-02-15 16:00:23,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32336.97 MB 2025-02-15 16:00:23,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.63 MB 2025-02-15 16:00:23,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:00:23,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-15 16:00:23,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 16:00:23,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37881.25 MB 2025-02-15 16:00:23,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:00:23,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:00:23,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 16:00:23,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33870.51 MB 2025-02-15 16:00:23,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34637.52 MB 2025-02-15 16:00:23,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 16:00:23,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-15 16:00:23,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:00:23,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 16:00:23,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35345.30 MB 2025-02-15 16:00:23,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:00:23,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:00:23,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:00:23,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35050.40 MB 2025-02-15 16:00:23,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35279.22 MB 2025-02-15 16:00:23,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-15 16:00:23,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:00:23,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:00:23,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:00:23,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35509.10 MB 2025-02-15 16:00:23,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:00:23,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:00:23,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.82 seconds 2025-02-15 16:00:23,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:23,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22722.09 MB 2025-02-15 16:00:23,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35479.95 MB 2025-02-15 16:00:23,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12757.85 MB 2025-02-15 16:00:23,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51476.69 MB 2025-02-15 16:00:23,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:00:23,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10466.89 MB 2025-02-15 16:00:23,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35509.10 MB 2025-02-15 16:00:24,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:00:24,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:00:24,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:00:24,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:24,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35479.95 MB 2025-02-15 16:00:24,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27721.15 MB 2025-02-15 16:00:24,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7758.80 MB 2025-02-15 16:00:24,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:00:24,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:00:24,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:00:24,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37987.31 MB 2025-02-15 16:00:24,122 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-15 16:00:24,122 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 16:00:24,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:00:24,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:00:24,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:00:24,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:00:24,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27721.15 MB 2025-02-15 16:00:24,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36146.10 MB 2025-02-15 16:00:24,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-15 16:00:24,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:00:24,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49385.83 MB 2025-02-15 16:00:24,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-15 16:00:24,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36146.10 MB 2025-02-15 16:00:24,297 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-15 16:00:24,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:24,298 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:00:24,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:24,299 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:00:24,304 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:00:24,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:00:24,305 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:00:24,305 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 16:01:11,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:11,741 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:01:11,747 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:01:11,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:11,751 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:01:11,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:11,752 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:01:14,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:01:14,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:01:14,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.97 seconds 2025-02-15 16:01:14,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:14,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19697.91 MB 2025-02-15 16:01:14,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20373.85 MB 2025-02-15 16:01:14,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-15 16:01:14,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57761.86 MB 2025-02-15 16:01:14,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22871.54 MB 2025-02-15 16:01:14,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34890.32 MB 2025-02-15 16:01:14,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29252.01 MB 2025-02-15 16:01:14,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:01:14,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:01:14,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:01:14,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:14,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20373.85 MB 2025-02-15 16:01:14,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20604.28 MB 2025-02-15 16:01:14,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.42 MB 2025-02-15 16:01:14,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22871.54 MB 2025-02-15 16:01:14,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24429.72 MB 2025-02-15 16:01:14,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1558.18 MB 2025-02-15 16:01:14,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22861.98 MB 2025-02-15 16:01:15,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:01:15,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:01:15,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-15 16:01:15,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20604.28 MB 2025-02-15 16:01:15,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20839.17 MB 2025-02-15 16:01:15,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-15 16:01:15,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24429.72 MB 2025-02-15 16:01:15,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24110.96 MB 2025-02-15 16:01:15,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -318.77 MB 2025-02-15 16:01:15,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24774.96 MB 2025-02-15 16:01:15,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:01:15,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:01:15,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:01:15,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20839.11 MB 2025-02-15 16:01:15,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21675.02 MB 2025-02-15 16:01:15,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-15 16:01:15,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24110.96 MB 2025-02-15 16:01:15,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24110.96 MB 2025-02-15 16:01:15,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:15,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22302.24 MB 2025-02-15 16:01:15,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:01:15,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:01:15,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 16:01:15,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21675.02 MB 2025-02-15 16:01:15,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22668.05 MB 2025-02-15 16:01:15,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 993.02 MB 2025-02-15 16:01:15,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24110.96 MB 2025-02-15 16:01:15,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26837.25 MB 2025-02-15 16:01:15,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2726.30 MB 2025-02-15 16:01:15,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25122.27 MB 2025-02-15 16:01:15,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:01:15,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:01:15,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 16:01:15,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20839.11 MB 2025-02-15 16:01:15,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22668.05 MB 2025-02-15 16:01:15,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1828.94 MB 2025-02-15 16:01:15,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24110.96 MB 2025-02-15 16:01:15,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26837.25 MB 2025-02-15 16:01:15,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2726.30 MB 2025-02-15 16:01:15,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25122.27 MB 2025-02-15 16:01:15,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:01:15,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:01:15,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 16:01:15,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23346.64 MB 2025-02-15 16:01:15,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23686.96 MB 2025-02-15 16:01:15,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-15 16:01:15,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26837.25 MB 2025-02-15 16:01:15,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 16:01:15,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-15 16:01:15,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24005.84 MB 2025-02-15 16:01:15,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:01:15,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:01:15,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:01:15,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23869.67 MB 2025-02-15 16:01:15,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24096.75 MB 2025-02-15 16:01:15,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.09 MB 2025-02-15 16:01:15,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 16:01:15,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 16:01:15,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:15,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24119.82 MB 2025-02-15 16:01:15,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:01:15,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:01:15,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.03 seconds 2025-02-15 16:01:15,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:15,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19032.45 MB 2025-02-15 16:01:15,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24297.82 MB 2025-02-15 16:01:15,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5265.37 MB 2025-02-15 16:01:15,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57761.86 MB 2025-02-15 16:01:15,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 16:01:15,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30742.15 MB 2025-02-15 16:01:15,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.82 MB 2025-02-15 16:01:16,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:01:16,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:01:16,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:01:16,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:16,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24297.82 MB 2025-02-15 16:01:16,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22985.00 MB 2025-02-15 16:01:16,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1312.83 MB 2025-02-15 16:01:16,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 16:01:16,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27019.71 MB 2025-02-15 16:01:16,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:16,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24398.32 MB 2025-02-15 16:01:16,069 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 16:01:16,069 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-15 16:01:16,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:01:16,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:01:16,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:01:16,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:16,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22985.00 MB 2025-02-15 16:01:16,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31424.02 MB 2025-02-15 16:01:16,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 16:01:16,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27019.71 MB 2025-02-15 16:01:16,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37509.66 MB 2025-02-15 16:01:16,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 16:01:16,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31424.02 MB 2025-02-15 16:01:16,238 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 16:01:16,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:16,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:01:16,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:16,241 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:01:16,245 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:01:16,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:16,247 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:01:16,247 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-15 16:01:27,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:27,031 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:01:27,036 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:01:27,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:27,039 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:01:27,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:27,040 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:01:46,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:01:46,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:01:46,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.21 seconds 2025-02-15 16:01:46,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:46,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26944.80 MB 2025-02-15 16:01:46,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31301.24 MB 2025-02-15 16:01:46,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4356.44 MB 2025-02-15 16:01:46,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50094.67 MB 2025-02-15 16:01:46,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43536.88 MB 2025-02-15 16:01:46,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6557.79 MB 2025-02-15 16:01:46,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40266.54 MB 2025-02-15 16:01:46,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:01:46,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:01:46,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 16:01:46,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:46,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31301.24 MB 2025-02-15 16:01:46,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27575.72 MB 2025-02-15 16:01:46,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3725.52 MB 2025-02-15 16:01:46,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43536.88 MB 2025-02-15 16:01:46,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49983.52 MB 2025-02-15 16:01:46,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6446.65 MB 2025-02-15 16:01:46,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44124.29 MB 2025-02-15 16:01:48,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:01:48,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:01:48,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 16:01:48,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27575.72 MB 2025-02-15 16:01:48,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28106.56 MB 2025-02-15 16:01:48,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 16:01:48,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49983.52 MB 2025-02-15 16:01:48,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 16:01:48,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10804.53 MB 2025-02-15 16:01:48,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32085.11 MB 2025-02-15 16:01:48,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:01:48,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:01:48,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:01:48,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28106.56 MB 2025-02-15 16:01:48,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.74 MB 2025-02-15 16:01:48,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 16:01:48,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:01:48,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39178.99 MB 2025-02-15 16:01:48,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:48,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31413.17 MB 2025-02-15 16:01:48,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:01:48,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:01:48,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 16:01:48,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29995.74 MB 2025-02-15 16:01:48,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32237.59 MB 2025-02-15 16:01:48,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 16:01:48,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:01:48,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-15 16:01:48,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 16:01:48,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37782.48 MB 2025-02-15 16:01:48,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:01:48,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:01:48,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 16:01:48,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28106.56 MB 2025-02-15 16:01:48,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32237.59 MB 2025-02-15 16:01:48,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 16:01:48,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39178.99 MB 2025-02-15 16:01:48,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-15 16:01:48,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-15 16:01:48,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37782.48 MB 2025-02-15 16:01:48,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:01:48,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:01:48,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 16:01:48,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33771.74 MB 2025-02-15 16:01:48,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34538.74 MB 2025-02-15 16:01:48,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 16:01:48,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-15 16:01:48,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:01:48,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 16:01:48,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35246.53 MB 2025-02-15 16:01:48,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:01:48,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:01:48,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:01:48,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34951.63 MB 2025-02-15 16:01:48,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35179.89 MB 2025-02-15 16:01:48,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-15 16:01:48,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:01:48,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:01:48,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:48,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35416.46 MB 2025-02-15 16:01:48,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:01:48,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:01:48,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.80 seconds 2025-02-15 16:01:48,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:48,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22655.90 MB 2025-02-15 16:01:48,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35380.57 MB 2025-02-15 16:01:48,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12724.67 MB 2025-02-15 16:01:48,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50094.67 MB 2025-02-15 16:01:48,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:01:48,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9084.86 MB 2025-02-15 16:01:48,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35416.46 MB 2025-02-15 16:01:49,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:01:49,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:01:49,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:01:49,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:49,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35380.57 MB 2025-02-15 16:01:49,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27654.79 MB 2025-02-15 16:01:49,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7725.78 MB 2025-02-15 16:01:49,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:01:49,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-15 16:01:49,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:01:49,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37887.32 MB 2025-02-15 16:01:49,131 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 16:01:49,131 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 16:01:49,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:01:49,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:01:49,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:01:49,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:01:49,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27654.79 MB 2025-02-15 16:01:49,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36077.12 MB 2025-02-15 16:01:49,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 16:01:49,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-15 16:01:49,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49383.74 MB 2025-02-15 16:01:49,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 16:01:49,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36077.12 MB 2025-02-15 16:01:49,300 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 16:01:49,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:49,302 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:01:49,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:49,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:01:49,307 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:01:49,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:01:49,309 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:01:49,309 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 16:03:05,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:05,673 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:03:05,681 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:03:05,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:05,688 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 220, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:03:05,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:05,690 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 220, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:03:09,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:03:09,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:03:09,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.54 seconds 2025-02-15 16:03:09,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:09,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19899.99 MB 2025-02-15 16:03:09,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20678.56 MB 2025-02-15 16:03:09,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 778.57 MB 2025-02-15 16:03:09,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61943.58 MB 2025-02-15 16:03:09,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23815.26 MB 2025-02-15 16:03:09,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38128.32 MB 2025-02-15 16:03:09,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.66 MB 2025-02-15 16:03:09,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:03:09,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:03:09,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:03:09,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:09,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20678.56 MB 2025-02-15 16:03:09,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21055.98 MB 2025-02-15 16:03:09,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 377.42 MB 2025-02-15 16:03:09,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23815.26 MB 2025-02-15 16:03:09,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25753.03 MB 2025-02-15 16:03:09,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1937.77 MB 2025-02-15 16:03:09,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23768.96 MB 2025-02-15 16:03:10,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:03:10,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:03:10,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.09 seconds 2025-02-15 16:03:10,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21055.98 MB 2025-02-15 16:03:10,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21347.94 MB 2025-02-15 16:03:10,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.96 MB 2025-02-15 16:03:10,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25753.03 MB 2025-02-15 16:03:10,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25375.54 MB 2025-02-15 16:03:10,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -377.49 MB 2025-02-15 16:03:10,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25311.60 MB 2025-02-15 16:03:10,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:03:10,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:03:10,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:03:10,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21347.94 MB 2025-02-15 16:03:10,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22386.93 MB 2025-02-15 16:03:10,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1038.99 MB 2025-02-15 16:03:10,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25375.54 MB 2025-02-15 16:03:10,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25375.54 MB 2025-02-15 16:03:10,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:03:10,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23166.52 MB 2025-02-15 16:03:10,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:03:10,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:03:10,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 16:03:10,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22386.93 MB 2025-02-15 16:03:10,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23621.63 MB 2025-02-15 16:03:10,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1234.70 MB 2025-02-15 16:03:10,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25375.54 MB 2025-02-15 16:03:10,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28496.10 MB 2025-02-15 16:03:10,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3120.56 MB 2025-02-15 16:03:10,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26675.15 MB 2025-02-15 16:03:10,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:03:10,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:03:10,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 16:03:10,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21347.94 MB 2025-02-15 16:03:10,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23621.63 MB 2025-02-15 16:03:10,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2273.69 MB 2025-02-15 16:03:10,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25375.54 MB 2025-02-15 16:03:10,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28496.10 MB 2025-02-15 16:03:10,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3120.56 MB 2025-02-15 16:03:10,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26675.15 MB 2025-02-15 16:03:10,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:03:10,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:03:10,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 16:03:10,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24465.08 MB 2025-02-15 16:03:10,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24886.93 MB 2025-02-15 16:03:10,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 421.85 MB 2025-02-15 16:03:10,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28496.10 MB 2025-02-15 16:03:10,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28722.59 MB 2025-02-15 16:03:10,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-15 16:03:10,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25276.22 MB 2025-02-15 16:03:10,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:03:10,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:03:10,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:03:10,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25114.03 MB 2025-02-15 16:03:10,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25325.25 MB 2025-02-15 16:03:10,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.22 MB 2025-02-15 16:03:10,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28722.59 MB 2025-02-15 16:03:10,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28724.69 MB 2025-02-15 16:03:10,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 16:03:10,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25372.20 MB 2025-02-15 16:03:10,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:03:10,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:03:10,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.92 seconds 2025-02-15 16:03:10,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19133.49 MB 2025-02-15 16:03:10,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25526.32 MB 2025-02-15 16:03:10,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6392.83 MB 2025-02-15 16:03:10,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61943.58 MB 2025-02-15 16:03:10,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28724.69 MB 2025-02-15 16:03:10,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33218.89 MB 2025-02-15 16:03:10,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25526.32 MB 2025-02-15 16:03:10,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:03:10,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:03:10,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:03:10,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20273.85 MB 2025-02-15 16:03:10,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23287.88 MB 2025-02-15 16:03:10,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 16:03:10,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28724.69 MB 2025-02-15 16:03:10,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28724.69 MB 2025-02-15 16:03:10,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:03:10,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23589.25 MB 2025-02-15 16:03:10,898 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 16:03:10,899 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for the video is 2.'] 2025-02-15 16:03:10,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:03:10,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:03:10,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:03:10,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:10,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23287.88 MB 2025-02-15 16:03:10,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31726.91 MB 2025-02-15 16:03:10,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 16:03:10,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28724.69 MB 2025-02-15 16:03:10,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39214.65 MB 2025-02-15 16:03:10,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 16:03:10,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31726.91 MB 2025-02-15 16:03:11,071 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 16:03:11,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:11,073 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:03:11,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:11,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:03:11,079 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:03:11,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:11,080 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:03:11,080 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for the video is 2.'] 2025-02-15 16:03:21,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:21,033 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:03:21,038 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:03:21,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:21,042 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1589, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:03:21,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:21,043 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1589, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:03:45,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:03:45,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:03:45,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.61 seconds 2025-02-15 16:03:45,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:45,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29439.40 MB 2025-02-15 16:03:45,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35062.78 MB 2025-02-15 16:03:45,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.38 MB 2025-02-15 16:03:45,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51799.65 MB 2025-02-15 16:03:45,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44803.56 MB 2025-02-15 16:03:45,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6996.10 MB 2025-02-15 16:03:45,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43894.41 MB 2025-02-15 16:03:45,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:03:45,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:03:45,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 16:03:45,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:45,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35062.78 MB 2025-02-15 16:03:45,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29436.85 MB 2025-02-15 16:03:45,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5625.93 MB 2025-02-15 16:03:45,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44803.56 MB 2025-02-15 16:03:45,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55515.81 MB 2025-02-15 16:03:45,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10712.25 MB 2025-02-15 16:03:45,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51171.75 MB 2025-02-15 16:03:47,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:03:47,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:03:47,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 16:03:47,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:47,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29436.85 MB 2025-02-15 16:03:47,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29967.69 MB 2025-02-15 16:03:47,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 16:03:47,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55515.81 MB 2025-02-15 16:03:47,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34984.69 MB 2025-02-15 16:03:47,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20531.12 MB 2025-02-15 16:03:47,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33946.24 MB 2025-02-15 16:03:47,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:03:47,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:03:47,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:03:47,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:47,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29967.69 MB 2025-02-15 16:03:47,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31856.87 MB 2025-02-15 16:03:47,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 16:03:47,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34984.69 MB 2025-02-15 16:03:47,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35928.41 MB 2025-02-15 16:03:47,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-15 16:03:47,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33274.30 MB 2025-02-15 16:03:47,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:03:47,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:03:47,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 16:03:47,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:47,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31856.87 MB 2025-02-15 16:03:47,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34098.72 MB 2025-02-15 16:03:47,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 16:03:47,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35928.41 MB 2025-02-15 16:03:47,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42062.58 MB 2025-02-15 16:03:47,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 16:03:47,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39643.01 MB 2025-02-15 16:03:47,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:03:47,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:03:47,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 16:03:47,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:47,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29967.69 MB 2025-02-15 16:03:47,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34098.72 MB 2025-02-15 16:03:47,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 16:03:47,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34984.69 MB 2025-02-15 16:03:47,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42062.58 MB 2025-02-15 16:03:47,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-15 16:03:47,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39643.01 MB 2025-02-15 16:03:48,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:03:48,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:03:48,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 16:03:48,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:48,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35632.87 MB 2025-02-15 16:03:48,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36399.87 MB 2025-02-15 16:03:48,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 16:03:48,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42062.58 MB 2025-02-15 16:03:48,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42477.81 MB 2025-02-15 16:03:48,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 16:03:48,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37107.66 MB 2025-02-15 16:03:48,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:03:48,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:03:48,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 16:03:48,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:48,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36812.76 MB 2025-02-15 16:03:48,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37040.53 MB 2025-02-15 16:03:48,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.77 MB 2025-02-15 16:03:48,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42477.81 MB 2025-02-15 16:03:48,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42477.81 MB 2025-02-15 16:03:48,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:03:48,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37240.49 MB 2025-02-15 16:03:48,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:03:48,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:03:48,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.27 seconds 2025-02-15 16:03:48,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:48,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23903.20 MB 2025-02-15 16:03:48,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37240.59 MB 2025-02-15 16:03:48,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13337.40 MB 2025-02-15 16:03:48,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51799.65 MB 2025-02-15 16:03:48,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42477.81 MB 2025-02-15 16:03:48,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9321.84 MB 2025-02-15 16:03:48,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37240.59 MB 2025-02-15 16:03:48,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:03:48,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:03:48,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-15 16:03:48,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:48,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37240.59 MB 2025-02-15 16:03:48,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.57 MB 2025-02-15 16:03:48,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8348.02 MB 2025-02-15 16:03:48,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42477.81 MB 2025-02-15 16:03:48,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42477.81 MB 2025-02-15 16:03:48,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:03:48,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39739.67 MB 2025-02-15 16:03:48,637 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-15 16:03:48,638 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 16:03:48,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:03:48,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:03:48,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:03:48,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:03:48,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28892.57 MB 2025-02-15 16:03:48,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37288.94 MB 2025-02-15 16:03:48,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.37 MB 2025-02-15 16:03:48,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42477.81 MB 2025-02-15 16:03:48,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46653.24 MB 2025-02-15 16:03:48,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-15 16:03:48,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37288.94 MB 2025-02-15 16:03:48,907 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-15 16:03:48,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:48,910 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:03:48,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:48,912 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:03:48,920 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:03:48,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:03:48,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:03:48,922 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 16:05:32,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:32,592 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:05:32,597 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:05:32,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:32,601 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:05:32,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:32,602 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:05:35,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:05:35,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:05:35,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.16 seconds 2025-02-15 16:05:35,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:35,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19788.50 MB 2025-02-15 16:05:35,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20510.44 MB 2025-02-15 16:05:35,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-15 16:05:35,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54999.91 MB 2025-02-15 16:05:35,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23817.36 MB 2025-02-15 16:05:35,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31182.55 MB 2025-02-15 16:05:35,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29487.17 MB 2025-02-15 16:05:35,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:05:35,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:05:35,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:05:35,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:35,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20510.44 MB 2025-02-15 16:05:35,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20784.38 MB 2025-02-15 16:05:35,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.94 MB 2025-02-15 16:05:35,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23817.36 MB 2025-02-15 16:05:35,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25174.21 MB 2025-02-15 16:05:35,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1356.86 MB 2025-02-15 16:05:35,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23222.81 MB 2025-02-15 16:05:36,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:05:36,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:05:36,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-15 16:05:36,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20784.38 MB 2025-02-15 16:05:36,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21040.52 MB 2025-02-15 16:05:36,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-15 16:05:36,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25174.21 MB 2025-02-15 16:05:36,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25174.21 MB 2025-02-15 16:05:36,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:05:36,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25040.01 MB 2025-02-15 16:05:36,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:05:36,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:05:36,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:05:36,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21040.45 MB 2025-02-15 16:05:36,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21951.93 MB 2025-02-15 16:05:36,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-15 16:05:36,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25174.21 MB 2025-02-15 16:05:36,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25174.21 MB 2025-02-15 16:05:36,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:05:36,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22635.84 MB 2025-02-15 16:05:36,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:05:36,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:05:36,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 16:05:36,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21951.93 MB 2025-02-15 16:05:36,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23033.66 MB 2025-02-15 16:05:36,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-15 16:05:36,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25174.21 MB 2025-02-15 16:05:36,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27460.11 MB 2025-02-15 16:05:36,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 16:05:36,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25708.74 MB 2025-02-15 16:05:36,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:05:36,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:05:36,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 16:05:36,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21040.45 MB 2025-02-15 16:05:36,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23033.66 MB 2025-02-15 16:05:36,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-15 16:05:36,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25174.21 MB 2025-02-15 16:05:36,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27460.11 MB 2025-02-15 16:05:36,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-15 16:05:36,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25708.74 MB 2025-02-15 16:05:36,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:05:36,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:05:36,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 16:05:36,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23773.59 MB 2025-02-15 16:05:36,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24143.67 MB 2025-02-15 16:05:36,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-15 16:05:36,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27460.11 MB 2025-02-15 16:05:36,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27657.24 MB 2025-02-15 16:05:36,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-15 16:05:36,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24487.85 MB 2025-02-15 16:05:36,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:05:36,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:05:36,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:05:36,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24342.90 MB 2025-02-15 16:05:36,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24569.66 MB 2025-02-15 16:05:36,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.77 MB 2025-02-15 16:05:36,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27657.24 MB 2025-02-15 16:05:36,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27657.24 MB 2025-02-15 16:05:36,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:05:36,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24601.82 MB 2025-02-15 16:05:36,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:05:36,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:05:36,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.32 seconds 2025-02-15 16:05:36,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:36,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.75 MB 2025-02-15 16:05:36,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24770.74 MB 2025-02-15 16:05:36,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5692.99 MB 2025-02-15 16:05:36,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54999.91 MB 2025-02-15 16:05:36,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27657.24 MB 2025-02-15 16:05:36,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27342.67 MB 2025-02-15 16:05:36,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24770.74 MB 2025-02-15 16:05:37,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:05:37,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:05:37,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 16:05:37,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:37,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20090.85 MB 2025-02-15 16:05:37,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23104.88 MB 2025-02-15 16:05:37,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 16:05:37,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27657.24 MB 2025-02-15 16:05:37,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27657.24 MB 2025-02-15 16:05:37,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:05:37,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23406.25 MB 2025-02-15 16:05:37,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 16:05:37,214 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 16:05:37,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:05:37,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:05:37,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:05:37,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:05:37,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23104.88 MB 2025-02-15 16:05:37,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31543.90 MB 2025-02-15 16:05:37,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 16:05:37,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27657.24 MB 2025-02-15 16:05:37,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38147.19 MB 2025-02-15 16:05:37,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 16:05:37,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31543.90 MB 2025-02-15 16:05:37,381 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 16:05:37,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:37,383 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:05:37,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:37,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:05:37,388 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:05:37,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:05:37,389 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:05:37,390 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 16:06:26,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:06:26,274 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:06:26,281 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:06:26,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:06:26,287 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1991, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:06:26,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:06:26,289 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1991, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:06:57,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:06:57,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:06:57,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.09 seconds 2025-02-15 16:06:57,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:57,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32240.60 MB 2025-02-15 16:06:57,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39287.03 MB 2025-02-15 16:06:57,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7046.43 MB 2025-02-15 16:06:57,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50732.20 MB 2025-02-15 16:06:57,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46225.42 MB 2025-02-15 16:06:57,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4506.78 MB 2025-02-15 16:06:57,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48281.06 MB 2025-02-15 16:06:57,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:06:57,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:06:57,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 16:06:57,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:57,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39287.03 MB 2025-02-15 16:06:57,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31526.73 MB 2025-02-15 16:06:57,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7760.31 MB 2025-02-15 16:06:57,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46225.42 MB 2025-02-15 16:06:57,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60825.80 MB 2025-02-15 16:06:57,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14600.37 MB 2025-02-15 16:06:57,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59113.83 MB 2025-02-15 16:06:59,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:06:59,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:06:59,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 16:06:59,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31526.73 MB 2025-02-15 16:06:59,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32057.57 MB 2025-02-15 16:06:59,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 16:06:59,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60825.80 MB 2025-02-15 16:06:59,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36400.27 MB 2025-02-15 16:06:59,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24425.53 MB 2025-02-15 16:06:59,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36037.15 MB 2025-02-15 16:06:59,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:06:59,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:06:59,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:06:59,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32057.57 MB 2025-02-15 16:06:59,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33946.74 MB 2025-02-15 16:06:59,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 16:06:59,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36400.27 MB 2025-02-15 16:06:59,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38287.70 MB 2025-02-15 16:06:59,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-15 16:06:59,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35364.17 MB 2025-02-15 16:06:59,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:06:59,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:06:59,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 16:06:59,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33946.74 MB 2025-02-15 16:06:59,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36188.60 MB 2025-02-15 16:06:59,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 16:06:59,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38287.70 MB 2025-02-15 16:06:59,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44421.87 MB 2025-02-15 16:06:59,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 16:06:59,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41732.88 MB 2025-02-15 16:06:59,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:06:59,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:06:59,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 16:06:59,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32057.57 MB 2025-02-15 16:06:59,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36188.60 MB 2025-02-15 16:06:59,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 16:06:59,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36400.27 MB 2025-02-15 16:06:59,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44421.87 MB 2025-02-15 16:06:59,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-15 16:06:59,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41732.88 MB 2025-02-15 16:06:59,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:06:59,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:06:59,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 16:06:59,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37722.74 MB 2025-02-15 16:06:59,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38489.74 MB 2025-02-15 16:06:59,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 16:06:59,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44421.87 MB 2025-02-15 16:06:59,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44837.11 MB 2025-02-15 16:06:59,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 16:06:59,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39197.53 MB 2025-02-15 16:06:59,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:06:59,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:06:59,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:06:59,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38902.63 MB 2025-02-15 16:06:59,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39131.59 MB 2025-02-15 16:06:59,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-15 16:06:59,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44837.11 MB 2025-02-15 16:06:59,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44837.11 MB 2025-02-15 16:06:59,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:06:59,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39352.07 MB 2025-02-15 16:06:59,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:06:59,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:06:59,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.62 seconds 2025-02-15 16:06:59,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:06:59,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25303.80 MB 2025-02-15 16:06:59,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39332.47 MB 2025-02-15 16:06:59,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14028.67 MB 2025-02-15 16:06:59,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50732.20 MB 2025-02-15 16:06:59,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44837.11 MB 2025-02-15 16:06:59,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5895.09 MB 2025-02-15 16:06:59,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39352.07 MB 2025-02-15 16:07:00,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:07:00,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:07:00,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:07:00,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:00,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39332.47 MB 2025-02-15 16:07:00,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30305.74 MB 2025-02-15 16:07:00,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9026.73 MB 2025-02-15 16:07:00,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44837.11 MB 2025-02-15 16:07:00,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44837.11 MB 2025-02-15 16:07:00,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:07:00,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41841.68 MB 2025-02-15 16:07:00,205 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-15 16:07:00,206 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 16:07:00,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:07:00,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:07:00,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:07:00,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:00,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30305.74 MB 2025-02-15 16:07:00,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38736.17 MB 2025-02-15 16:07:00,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.43 MB 2025-02-15 16:07:00,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44837.11 MB 2025-02-15 16:07:00,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49029.32 MB 2025-02-15 16:07:00,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-15 16:07:00,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38736.17 MB 2025-02-15 16:07:00,370 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-15 16:07:00,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:00,372 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:07:00,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:00,373 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:07:00,377 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:07:00,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:00,378 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:07:00,379 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-15 16:07:18,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:18,357 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 16:07:18,365 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 16:07:18,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:18,372 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 16:07:18,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:18,374 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 16:07:37,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 16:07:37,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 871 2025-02-15 16:07:37,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.91 seconds 2025-02-15 16:07:37,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:37,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26735.75 MB 2025-02-15 16:07:37,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30986.68 MB 2025-02-15 16:07:37,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-15 16:07:37,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57409.54 MB 2025-02-15 16:07:37,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35030.83 MB 2025-02-15 16:07:37,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22378.71 MB 2025-02-15 16:07:37,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39831.81 MB 2025-02-15 16:07:37,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 16:07:37,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 16:07:37,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 16:07:37,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:37,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30986.68 MB 2025-02-15 16:07:37,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27420.81 MB 2025-02-15 16:07:37,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3565.87 MB 2025-02-15 16:07:37,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35030.83 MB 2025-02-15 16:07:37,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42932.90 MB 2025-02-15 16:07:37,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7902.07 MB 2025-02-15 16:07:37,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40986.36 MB 2025-02-15 16:07:39,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 16:07:39,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 892 2025-02-15 16:07:39,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 16:07:39,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27420.81 MB 2025-02-15 16:07:39,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27951.65 MB 2025-02-15 16:07:39,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 16:07:39,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42932.90 MB 2025-02-15 16:07:39,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32904.31 MB 2025-02-15 16:07:39,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10028.58 MB 2025-02-15 16:07:39,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31930.20 MB 2025-02-15 16:07:39,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 16:07:39,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 933 2025-02-15 16:07:39,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 16:07:39,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27951.65 MB 2025-02-15 16:07:39,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29840.83 MB 2025-02-15 16:07:39,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-15 16:07:39,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32904.31 MB 2025-02-15 16:07:39,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32904.31 MB 2025-02-15 16:07:39,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:07:39,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31258.26 MB 2025-02-15 16:07:39,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 16:07:39,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 951 2025-02-15 16:07:39,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:07:39,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29840.83 MB 2025-02-15 16:07:39,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32082.68 MB 2025-02-15 16:07:39,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 16:07:39,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32904.31 MB 2025-02-15 16:07:39,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39510.34 MB 2025-02-15 16:07:39,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 16:07:39,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37627.57 MB 2025-02-15 16:07:39,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 16:07:39,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 928 2025-02-15 16:07:39,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 16:07:39,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27951.65 MB 2025-02-15 16:07:39,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32082.68 MB 2025-02-15 16:07:39,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-15 16:07:39,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32904.31 MB 2025-02-15 16:07:39,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39510.34 MB 2025-02-15 16:07:39,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 16:07:39,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37627.57 MB 2025-02-15 16:07:39,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 16:07:39,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1094 2025-02-15 16:07:39,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 16:07:39,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33616.83 MB 2025-02-15 16:07:39,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34383.83 MB 2025-02-15 16:07:39,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 16:07:39,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39510.34 MB 2025-02-15 16:07:39,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39925.58 MB 2025-02-15 16:07:39,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-15 16:07:39,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35091.62 MB 2025-02-15 16:07:39,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 16:07:39,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1395 2025-02-15 16:07:39,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:07:39,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34796.72 MB 2025-02-15 16:07:39,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35026.78 MB 2025-02-15 16:07:39,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.07 MB 2025-02-15 16:07:39,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39925.58 MB 2025-02-15 16:07:39,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39925.58 MB 2025-02-15 16:07:39,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:07:39,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35251.79 MB 2025-02-15 16:07:39,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 16:07:39,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 16:07:39,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.43 seconds 2025-02-15 16:07:39,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:39,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22551.37 MB 2025-02-15 16:07:39,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35227.86 MB 2025-02-15 16:07:39,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12676.48 MB 2025-02-15 16:07:39,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57409.54 MB 2025-02-15 16:07:39,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39925.58 MB 2025-02-15 16:07:39,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17483.96 MB 2025-02-15 16:07:39,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35251.79 MB 2025-02-15 16:07:40,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 16:07:40,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 16:07:40,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 16:07:40,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:40,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35227.86 MB 2025-02-15 16:07:40,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27556.37 MB 2025-02-15 16:07:40,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7671.49 MB 2025-02-15 16:07:40,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39925.58 MB 2025-02-15 16:07:40,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39925.58 MB 2025-02-15 16:07:40,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 16:07:40,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37739.52 MB 2025-02-15 16:07:40,092 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 16:07:40,092 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-15 16:07:40,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 16:07:40,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 16:07:40,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 16:07:40,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 16:07:40,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27556.37 MB 2025-02-15 16:07:40,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35995.39 MB 2025-02-15 16:07:40,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 16:07:40,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39925.58 MB 2025-02-15 16:07:40,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48316.28 MB 2025-02-15 16:07:40,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-15 16:07:40,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35995.39 MB 2025-02-15 16:07:40,257 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 16:07:40,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:40,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 16:07:40,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:40,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 16:07:40,264 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 16:07:40,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 16:07:40,265 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 16:07:40,265 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.']